[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:57:28 dist-test-slave-sb6p krb5kdc[399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:57:28 dist-test-slave-sb6p krb5kdc[399](info): set up 2 sockets
Sep 24 02:57:28 dist-test-slave-sb6p krb5kdc[399](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260924 02:57:29.723532 392 mini_kdc.cc:121] Time spent starting KDC: real 2.369s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
W20260924 02:57:29.868009 392 mini_kdc.cc:253] Time spent creating user principal test-admin: real 0.144s user 0.000s sys 0.002s
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
W20260924 02:57:29.973232 392 mini_kdc.cc:253] Time spent creating user principal test-user: real 0.105s user 0.000s sys 0.001s
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
W20260924 02:57:30.092679 392 mini_kdc.cc:253] Time spent creating user principal joe-interloper: real 0.119s user 0.001s sys 0.000s
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/test-user.keytab.
W20260924 02:57:30.410858 392 mini_kdc.cc:302] Time spent creating keytab for test-user: real 0.318s user 0.001s sys 0.000s
Sep 24 02:57:30 dist-test-slave-sb6p krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:57:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:57:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
W20260924 02:57:31.556581 392 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.0.98.62: real 1.126s user 0.002s sys 0.000s
I20260924 02:57:31.557042 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37319
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37319
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:31.634166 412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:31.634356 412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:31.634377 412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:31.635551 412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:31.635593 412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:31.635607 412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:31.635617 412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:31.635628 412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:57:31.637156 412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37319
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37319
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:31.637393 412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:31.637624 412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 02:57:31.642555 412 server_base.cc:1034] running on GCE node
W20260924 02:57:31.642673 420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:31.642640 417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:31.642710 418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:31.643049 412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:31.643317 412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:31.644457 412 hybrid_clock.cc:648] HybridClock initialized: now 1790218651644438 us; error 30 us; skew 500 ppm
Sep 24 02:57:31 dist-test-slave-sb6p krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:31.646790 412 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:57:31.647214 412 webserver.cc:460] Webserver started at http://127.0.98.62:38587/ using document root <none> and password file <none>
I20260924 02:57:31.647451 412 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:31.647501 412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:31.647634 412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:31.648674 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "0b9f1fb9e4b347269bc03369729fd17e"
format_stamp: "Formatted at 2026-09-24 02:57:31 on dist-test-slave-sb6p"
server_key: "f2c22cb9460a826af1694550bc44d033"
server_key_iv: "e05627576788b5c154a97ea5c25c92c9"
server_key_version: "encryptionkey@0"
I20260924 02:57:31.648998 412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "0b9f1fb9e4b347269bc03369729fd17e"
format_stamp: "Formatted at 2026-09-24 02:57:31 on dist-test-slave-sb6p"
server_key: "f2c22cb9460a826af1694550bc44d033"
server_key_iv: "e05627576788b5c154a97ea5c25c92c9"
server_key_version: "encryptionkey@0"
I20260924 02:57:31.650285 412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 02:57:31.651084 427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:31.651234 412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:31.651314 412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "0b9f1fb9e4b347269bc03369729fd17e"
format_stamp: "Formatted at 2026-09-24 02:57:31 on dist-test-slave-sb6p"
server_key: "f2c22cb9460a826af1694550bc44d033"
server_key_iv: "e05627576788b5c154a97ea5c25c92c9"
server_key_version: "encryptionkey@0"
I20260924 02:57:31.651377 412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:31.663038 412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:31.665033 412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:31.665177 412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:31.668917 412 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37319
I20260924 02:57:31.668946 479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37319 every 8 connection(s)
I20260924 02:57:31.669250 412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:57:31.670053 480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:31.671360 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 412
I20260924 02:57:31.671465 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:57:31.672233 392 external_mini_cluster.cc:1428] Setting key d8e806936c20a840db436f7a966efa19
I20260924 02:57:31.672762 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e: Bootstrap starting.
I20260924 02:57:31.673367 480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:31.673695 480 log.cc:824] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:31.674405 480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e: No bootstrap required, opened a new log
I20260924 02:57:31.676385 480 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9f1fb9e4b347269bc03369729fd17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37319 } }
I20260924 02:57:31.676524 480 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:31.676602 480 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0b9f1fb9e4b347269bc03369729fd17e, State: Initialized, Role: FOLLOWER
I20260924 02:57:31.676734 480 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9f1fb9e4b347269bc03369729fd17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37319 } }
I20260924 02:57:31.676813 480 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:57:31.676848 480 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:57:31.676892 480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:31.677836 480 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9f1fb9e4b347269bc03369729fd17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37319 } }
I20260924 02:57:31.677951 480 leader_election.cc:302] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0b9f1fb9e4b347269bc03369729fd17e; no voters:
I20260924 02:57:31.678100 480 leader_election.cc:288] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:57:31.678122 485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:31.678252 485 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [term 1 LEADER]: Becoming Leader. State: Replica: 0b9f1fb9e4b347269bc03369729fd17e, State: Running, Role: LEADER
I20260924 02:57:31.678340 480 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:31.678369 485 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9f1fb9e4b347269bc03369729fd17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37319 } }
I20260924 02:57:31.679679 486 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0b9f1fb9e4b347269bc03369729fd17e. Latest consensus state: current_term: 1 leader_uuid: "0b9f1fb9e4b347269bc03369729fd17e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9f1fb9e4b347269bc03369729fd17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37319 } } }
I20260924 02:57:31.679795 486 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:31.679673 487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0b9f1fb9e4b347269bc03369729fd17e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0b9f1fb9e4b347269bc03369729fd17e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37319 } } }
I20260924 02:57:31.680101 487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:31.680160 500 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:31.680615 500 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:57:31.682000 500 catalog_manager.cc:1347] Generated new cluster ID: 537a439555844401890946e1fa9651ff
I20260924 02:57:31.682047 500 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 02:57:31 dist-test-slave-sb6p krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:31.689576 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:31.673162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.0.98.62:37319)
0924 02:57:31.673324 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:31.673333 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:31.673566 (+ 233us) server_negotiation.cc:407] Connection header received
0924 02:57:31.676357 (+ 2791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:31.676363 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:31.676406 (+ 43us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:31.676509 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:31.678534 (+ 2025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:31.679729 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:31.680957 (+ 1228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:31.681345 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:31.681440 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:31.685300 (+ 3860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:31.685304 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:31.685304 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:31.685315 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:31.687708 (+ 2393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:31.688190 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:31.688193 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:31.688194 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:31.688299 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:31.688557 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:31.688557 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:31.688558 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:31.688823 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:31.688963 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:31.689240 (+ 277us) server_negotiation.cc:299] Negotiation successful
0924 02:57:31.689367 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260924 02:57:31.695827 500 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:57:31.696353 500 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:31.698767 500 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0b9f1fb9e4b347269bc03369729fd17e: Generated new TSK 0
I20260924 02:57:31.698910 500 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 02:57:31.729064 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37319
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:31.842738 506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:31.842999 506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:31.843034 506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:31.844374 506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:31.844420 506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:31.844446 506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:57:31.845805 506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37319
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:31.846001 506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:31.846182 506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:31.850775 512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:31.850780 511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:31.850895 514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:31.850973 506 server_base.cc:1034] running on GCE node
I20260924 02:57:31.851154 506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:31.851334 506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:31.852463 506 hybrid_clock.cc:648] HybridClock initialized: now 1790218651852448 us; error 26 us; skew 500 ppm
Sep 24 02:57:31 dist-test-slave-sb6p krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:31.854763 506 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 02:57:31.855280 506 webserver.cc:460] Webserver started at http://127.0.98.1:41507/ using document root <none> and password file <none>
I20260924 02:57:31.855489 506 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:31.855530 506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:31.855639 506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:31.856956 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "07b1acff110b40c1a91224baf6dddede"
format_stamp: "Formatted at 2026-09-24 02:57:31 on dist-test-slave-sb6p"
server_key: "8f5872963f2302fc1a68a19e8a86c1d4"
server_key_iv: "ce798e96ea20cdbe2f532bee8107a7f6"
server_key_version: "encryptionkey@0"
I20260924 02:57:31.857338 506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "07b1acff110b40c1a91224baf6dddede"
format_stamp: "Formatted at 2026-09-24 02:57:31 on dist-test-slave-sb6p"
server_key: "8f5872963f2302fc1a68a19e8a86c1d4"
server_key_iv: "ce798e96ea20cdbe2f532bee8107a7f6"
server_key_version: "encryptionkey@0"
I20260924 02:57:31.859452 506 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260924 02:57:31.860224 521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:31.860386 506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:31.860488 506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "07b1acff110b40c1a91224baf6dddede"
format_stamp: "Formatted at 2026-09-24 02:57:31 on dist-test-slave-sb6p"
server_key: "8f5872963f2302fc1a68a19e8a86c1d4"
server_key_iv: "ce798e96ea20cdbe2f532bee8107a7f6"
server_key_version: "encryptionkey@0"
I20260924 02:57:31.860554 506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:31.870810 506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:31.872442 506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:31.872562 506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:31.872756 506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:31.873023 506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:31.873054 506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:31.873080 506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:31.873111 506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:31.878468 506 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40587
I20260924 02:57:31.878535 634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40587 every 8 connection(s)
I20260924 02:57:31.878825 506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:57:31.883940 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 506
I20260924 02:57:31.884037 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:57:31.884194 392 external_mini_cluster.cc:1428] Setting key a57258bc150928d630428bb4a0acebfe
Sep 24 02:57:31 dist-test-slave-sb6p krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:31.887481 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:31.879455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47781 (local address 127.0.98.62:37319)
0924 02:57:31.879502 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:31.879504 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:31.879746 (+ 242us) server_negotiation.cc:407] Connection header received
0924 02:57:31.880258 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:31.880259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:31.880276 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:31.880326 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:31.881207 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:31.881701 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:31.882717 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:31.882941 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:31.882997 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:31.885058 (+ 2061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:31.885059 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:31.885060 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:31.885066 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:31.886208 (+ 1142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:31.886650 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:31.886650 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:31.886650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:31.886697 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:31.886931 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:31.886931 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:31.886932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:31.887022 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:31.887065 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:31.887355 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 02:57:31.887419 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 02:57:31.887687 637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:31.879437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37319 (local address 127.0.98.1:47781)
0924 02:57:31.879600 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:31.879667 (+ 67us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:31.880165 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:31.880392 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:31.880396 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:31.880937 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:31.881118 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:31.881123 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:31.881779 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:31.881780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:31.882629 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:31.882631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:31.883009 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:31.883010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:31.883131 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:31.883523 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:31.883530 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:31.884977 (+ 1447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:31.886278 (+ 1301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:31.886285 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:31.886287 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:31.886583 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:31.886752 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:31.886753 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:31.886753 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:31.886878 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:31.887077 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:31.887082 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:31.887233 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 02:57:31.887333 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 02:57:31.887483 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":83,"threads_started":1}
I20260924 02:57:31.888203 635 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37319
I20260924 02:57:31.888294 635 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:31.888459 635 heartbeater.cc:511] Master 127.0.98.62:37319 requested a full tablet report, sending...
I20260924 02:57:31.888995 444 ts_manager.cc:194] Registered new tserver with Master: 07b1acff110b40c1a91224baf6dddede (127.0.98.1:40587)
I20260924 02:57:31.889916 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:47781
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 02:57:31.907316 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37319
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:31.987527 640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:31.987730 640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:31.987761 640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:31.988852 640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:31.988888 640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:31.988922 640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:57:31.990310 640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37319
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:31.990547 640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:31.990742 640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:31.995194 645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:31.995194 648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:31.995273 646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:31.995307 640 server_base.cc:1034] running on GCE node
I20260924 02:57:31.995543 640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:31.995792 640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:31.996970 640 hybrid_clock.cc:648] HybridClock initialized: now 1790218651996954 us; error 33 us; skew 500 ppm
Sep 24 02:57:31 dist-test-slave-sb6p krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:31.999150 640 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 02:57:31.999540 640 webserver.cc:460] Webserver started at http://127.0.98.2:42955/ using document root <none> and password file <none>
I20260924 02:57:31.999744 640 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:31.999785 640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:31.999908 640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:32.000962 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "33c02bcaba15457d8838fe1d671adf98"
format_stamp: "Formatted at 2026-09-24 02:57:31 on dist-test-slave-sb6p"
server_key: "433200334e5150d97dcbd88c7406d48b"
server_key_iv: "b5ad311b85b3d58b3e367182cd3cbe96"
server_key_version: "encryptionkey@0"
I20260924 02:57:32.001278 640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "33c02bcaba15457d8838fe1d671adf98"
format_stamp: "Formatted at 2026-09-24 02:57:31 on dist-test-slave-sb6p"
server_key: "433200334e5150d97dcbd88c7406d48b"
server_key_iv: "b5ad311b85b3d58b3e367182cd3cbe96"
server_key_version: "encryptionkey@0"
I20260924 02:57:32.002467 640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 02:57:32.003173 655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:32.003316 640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:32.003392 640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "33c02bcaba15457d8838fe1d671adf98"
format_stamp: "Formatted at 2026-09-24 02:57:31 on dist-test-slave-sb6p"
server_key: "433200334e5150d97dcbd88c7406d48b"
server_key_iv: "b5ad311b85b3d58b3e367182cd3cbe96"
server_key_version: "encryptionkey@0"
I20260924 02:57:32.003458 640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:32.022112 640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:32.023912 640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:32.024044 640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:32.024240 640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:32.024523 640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:32.024560 640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:32.024598 640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:32.024623 640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:32.030195 640 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41331
I20260924 02:57:32.030756 640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 02:57:32.030746 768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41331 every 8 connection(s)
I20260924 02:57:32.031132 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 640
I20260924 02:57:32.031226 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:57:32.031383 392 external_mini_cluster.cc:1428] Setting key 69182a19647b7af357e1f2a65e2cfea1
Sep 24 02:57:32 dist-test-slave-sb6p krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 02:57:32.040045 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:32.031901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48699 (local address 127.0.98.62:37319)
0924 02:57:32.031941 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:32.031943 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:32.032213 (+ 270us) server_negotiation.cc:407] Connection header received
0924 02:57:32.032653 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:32.032653 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:32.032664 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:32.032729 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:32.033604 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:32.034197 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:32.035169 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:32.035463 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:32.035536 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:32.037439 (+ 1903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:32.037440 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:32.037441 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:32.037448 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:32.038951 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:32.039310 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:32.039310 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:32.039311 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:32.039357 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:32.039533 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:32.039533 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:32.039533 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:32.039633 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:32.039680 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:32.039920 (+ 240us) server_negotiation.cc:299] Negotiation successful
0924 02:57:32.039983 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 02:57:32.040226 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:32.031923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37319 (local address 127.0.98.2:48699)
0924 02:57:32.032090 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:32.032132 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:32.032577 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:32.032783 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:32.032786 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:32.033326 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:32.033515 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:32.033520 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:32.034263 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:32.034263 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:32.035078 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:32.035080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:32.035537 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:32.035537 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:32.035653 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:32.036106 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:32.036114 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:32.037352 (+ 1238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:32.039017 (+ 1665us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:32.039022 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:32.039024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:32.039247 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:32.039405 (+ 158us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:32.039406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:32.039406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:32.039491 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:32.039678 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:32.039681 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:32.039798 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 02:57:32.039903 (+ 105us) client_negotiation.cc:240] Negotiation successful
0924 02:57:32.040021 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260924 02:57:32.040676 769 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37319
I20260924 02:57:32.040758 769 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:32.040918 769 heartbeater.cc:511] Master 127.0.98.62:37319 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 02:57:32.041268 444 ts_manager.cc:194] Registered new tserver with Master: 33c02bcaba15457d8838fe1d671adf98 (127.0.98.2:41331)
I20260924 02:57:32.041644 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:48699
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 02:57:32.052934 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37319
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:32.132027 774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:32.132215 774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:32.132236 774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:32.133350 774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:32.133391 774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:32.133417 774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:57:32.134829 774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37319
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:32.135044 774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:32.135241 774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:32.140949 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:32.141048 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:32.141081 774 server_base.cc:1034] running on GCE node
W20260924 02:57:32.140954 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:32.141399 774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:32.141638 774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:32.142719 774 hybrid_clock.cc:648] HybridClock initialized: now 1790218652142700 us; error 37 us; skew 500 ppm
Sep 24 02:57:32 dist-test-slave-sb6p krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:32.145571 774 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 02:57:32.145982 774 webserver.cc:460] Webserver started at http://127.0.98.3:46349/ using document root <none> and password file <none>
I20260924 02:57:32.146185 774 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:32.146229 774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:32.146349 774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:32.147406 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "ce82e07dac3a47018de32c7b0eb60d78"
format_stamp: "Formatted at 2026-09-24 02:57:32 on dist-test-slave-sb6p"
server_key: "43e97cfe58b95d3964311d41f34be49c"
server_key_iv: "8941a20ced4ff37c7a842ee285666c4a"
server_key_version: "encryptionkey@0"
I20260924 02:57:32.147744 774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ce82e07dac3a47018de32c7b0eb60d78"
format_stamp: "Formatted at 2026-09-24 02:57:32 on dist-test-slave-sb6p"
server_key: "43e97cfe58b95d3964311d41f34be49c"
server_key_iv: "8941a20ced4ff37c7a842ee285666c4a"
server_key_version: "encryptionkey@0"
I20260924 02:57:32.148896 774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:57:32.149515 789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:32.149652 774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:32.149729 774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "ce82e07dac3a47018de32c7b0eb60d78"
format_stamp: "Formatted at 2026-09-24 02:57:32 on dist-test-slave-sb6p"
server_key: "43e97cfe58b95d3964311d41f34be49c"
server_key_iv: "8941a20ced4ff37c7a842ee285666c4a"
server_key_version: "encryptionkey@0"
I20260924 02:57:32.149792 774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:32.174224 774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:32.176302 774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:32.176431 774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:32.176640 774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:32.176929 774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:32.176968 774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:32.176998 774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:32.177022 774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:32.183321 774 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37601
I20260924 02:57:32.183362 902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37601 every 8 connection(s)
I20260924 02:57:32.183668 774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 02:57:32.187116 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 774
I20260924 02:57:32.187212 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:57:32.187361 392 external_mini_cluster.cc:1428] Setting key 69c356d4729377134e1b376bd961ceb6
Sep 24 02:57:32 dist-test-slave-sb6p krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:32.192277 482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:32.184245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55795 (local address 127.0.98.62:37319)
0924 02:57:32.184284 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:32.184287 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:32.184475 (+ 188us) server_negotiation.cc:407] Connection header received
0924 02:57:32.184949 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:32.184950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:32.184963 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:32.185013 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:32.185908 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:32.186402 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:32.187681 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:32.187882 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:32.187938 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:32.190002 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:32.190003 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:32.190003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:32.190011 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:32.191092 (+ 1081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:32.191436 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:32.191437 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:32.191438 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:32.191485 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:32.191694 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:32.191695 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:32.191695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:32.191798 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:32.191853 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:32.192116 (+ 263us) server_negotiation.cc:299] Negotiation successful
0924 02:57:32.192191 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 02:57:32.192444 905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:32.184233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37319 (local address 127.0.98.3:55795)
0924 02:57:32.184361 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:32.184396 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:32.184859 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:32.185090 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:32.185093 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:32.185630 (+ 537us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:32.185816 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:32.185821 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:32.186480 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:32.186481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:32.187601 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:32.187604 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:32.187947 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:32.187948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:32.188078 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:32.188651 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:32.188659 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:32.189914 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:32.191162 (+ 1248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:32.191168 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:32.191170 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:32.191373 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:32.191538 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:32.191538 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:32.191539 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:32.191639 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:32.191851 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:32.191857 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:32.192002 (+ 145us) client_negotiation.cc:769] Sending connection context
0924 02:57:32.192094 (+ 92us) client_negotiation.cc:240] Negotiation successful
0924 02:57:32.192244 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":64,"threads_started":1}
I20260924 02:57:32.192828 903 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37319
I20260924 02:57:32.192917 903 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:32.193078 903 heartbeater.cc:511] Master 127.0.98.62:37319 requested a full tablet report, sending...
I20260924 02:57:32.193397 444 ts_manager.cc:194] Registered new tserver with Master: ce82e07dac3a47018de32c7b0eb60d78 (127.0.98.3:37601)
I20260924 02:57:32.193737 444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55795
I20260924 02:57:32.199184 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 02:57:32 dist-test-slave-sb6p krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218652, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 02:57:32 dist-test-slave-sb6p krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218652, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 02:57:32.220371 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:32.212950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39916 (local address 127.0.98.1:40587)
0924 02:57:32.213135 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:32.213141 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:32.213150 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:57:32.213173 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:32.213174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:32.213186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:32.213249 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:32.213771 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:32.214483 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:32.215213 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:32.215530 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:32.215589 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:32.217774 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:32.217782 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:32.217783 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:32.217790 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:32.219327 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:32.219639 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:32.219642 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:32.219643 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:32.219711 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:32.219867 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:32.219868 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:32.219868 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:32.219998 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:32.220125 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:32.220191 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 02:57:32.220265 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":151,"thread_start_us":97,"threads_started":1}
W20260924 02:57:32.220743 549 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39916
Sep 24 02:57:32 dist-test-slave-sb6p krb5kdc[399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 02:57:32 dist-test-slave-sb6p krb5kdc[399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218652, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 02:57:32.241133 910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:32.234721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39932 (local address 127.0.98.1:40587)
0924 02:57:32.234773 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:32.234776 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:32.234815 (+ 39us) server_negotiation.cc:407] Connection header received
0924 02:57:32.234893 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:32.234894 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:32.234904 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:32.234950 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:32.235512 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:32.235987 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:32.236674 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:32.236873 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:32.236926 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:32.238943 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:32.238944 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:32.238944 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:32.238953 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:32.240142 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:32.240418 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:32.240419 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:32.240419 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:32.240485 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:32.240665 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:32.240666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:32.240666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:32.240772 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:32.240880 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:32.240927 (+ 47us) server_negotiation.cc:299] Negotiation successful
0924 02:57:32.241042 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 02:57:32.241636 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 506
I20260924 02:57:32.246006 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 640
I20260924 02:57:32.250706 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 774
I20260924 02:57:32.255005 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 412
2026-09-24T02:57:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4925 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:57:32 dist-test-slave-sb6p krb5kdc[917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:57:32 dist-test-slave-sb6p krb5kdc[917](info): set up 2 sockets
Sep 24 02:57:32 dist-test-slave-sb6p krb5kdc[917](info): commencing operation
krb5kdc: starting...
W20260924 02:57:33.209723 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:57:33 dist-test-slave-sb6p krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:57:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:57:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 02:57:33.314664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39689
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39689
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:33.391608 930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:33.391799 930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:33.391819 930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:33.392921 930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:33.392961 930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:33.392973 930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:33.392984 930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:33.392997 930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:57:33.394424 930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39689
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39689
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:33.394673 930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:33.394873 930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:33.399358 935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:33.399350 938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:33.399353 936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:33.399540 930 server_base.cc:1034] running on GCE node
I20260924 02:57:33.399732 930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:33.399971 930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:33.401120 930 hybrid_clock.cc:648] HybridClock initialized: now 1790218653401110 us; error 27 us; skew 500 ppm
Sep 24 02:57:33 dist-test-slave-sb6p krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:33.406484 930 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:57:33.406916 930 webserver.cc:460] Webserver started at http://127.0.98.62:35775/ using document root <none> and password file <none>
I20260924 02:57:33.407126 930 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:33.407174 930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:33.407292 930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:33.408345 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "9032f57107a2474e9215bec4b7a4de29"
format_stamp: "Formatted at 2026-09-24 02:57:33 on dist-test-slave-sb6p"
server_key: "5383ae9a522ac5b214e9a3a59127d35d"
server_key_iv: "5290daea81afc978056769e2c903ffda"
server_key_version: "encryptionkey@0"
I20260924 02:57:33.408658 930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "9032f57107a2474e9215bec4b7a4de29"
format_stamp: "Formatted at 2026-09-24 02:57:33 on dist-test-slave-sb6p"
server_key: "5383ae9a522ac5b214e9a3a59127d35d"
server_key_iv: "5290daea81afc978056769e2c903ffda"
server_key_version: "encryptionkey@0"
I20260924 02:57:33.409823 930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 02:57:33.410459 945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:33.410636 930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:33.410717 930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "9032f57107a2474e9215bec4b7a4de29"
format_stamp: "Formatted at 2026-09-24 02:57:33 on dist-test-slave-sb6p"
server_key: "5383ae9a522ac5b214e9a3a59127d35d"
server_key_iv: "5290daea81afc978056769e2c903ffda"
server_key_version: "encryptionkey@0"
I20260924 02:57:33.410776 930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:33.423810 930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:33.425496 930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:33.425622 930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:33.429230 930 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39689
I20260924 02:57:33.429273 997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39689 every 8 connection(s)
I20260924 02:57:33.429566 930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:57:33.430469 998 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:33.432683 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29: Bootstrap starting.
I20260924 02:57:33.433226 998 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:33.433485 998 log.cc:824] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:33.434140 998 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29: No bootstrap required, opened a new log
I20260924 02:57:33.435752 998 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9032f57107a2474e9215bec4b7a4de29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } }
I20260924 02:57:33.435885 998 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:33.435907 998 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9032f57107a2474e9215bec4b7a4de29, State: Initialized, Role: FOLLOWER
I20260924 02:57:33.436049 998 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9032f57107a2474e9215bec4b7a4de29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } }
I20260924 02:57:33.436115 998 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:57:33.436138 998 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:57:33.436182 998 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:33.436667 998 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9032f57107a2474e9215bec4b7a4de29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } }
I20260924 02:57:33.436753 998 leader_election.cc:302] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9032f57107a2474e9215bec4b7a4de29; no voters:
I20260924 02:57:33.436899 998 leader_election.cc:288] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:57:33.436941 1001 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:33.437139 998 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:33.437265 1001 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [term 1 LEADER]: Becoming Leader. State: Replica: 9032f57107a2474e9215bec4b7a4de29, State: Running, Role: LEADER
I20260924 02:57:33.437361 1001 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9032f57107a2474e9215bec4b7a4de29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } }
I20260924 02:57:33.437662 1002 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9032f57107a2474e9215bec4b7a4de29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9032f57107a2474e9215bec4b7a4de29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } } }
I20260924 02:57:33.437677 1003 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9032f57107a2474e9215bec4b7a4de29. Latest consensus state: current_term: 1 leader_uuid: "9032f57107a2474e9215bec4b7a4de29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9032f57107a2474e9215bec4b7a4de29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39689 } } }
I20260924 02:57:33.437760 1003 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:33.437749 1002 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:33.437996 1010 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:33.438661 1010 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:57:33.438719 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 930
I20260924 02:57:33.438804 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:57:33.438959 392 external_mini_cluster.cc:1428] Setting key 79a984b07800ef983ec3898fbb0df977
I20260924 02:57:33.440757 1010 catalog_manager.cc:1347] Generated new cluster ID: 5e72970ca1d84a84b987c0750612dc25
I20260924 02:57:33.440801 1010 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 02:57:33 dist-test-slave-sb6p krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:33.449934 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:33.439382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44240 (local address 127.0.98.62:39689)
0924 02:57:33.439566 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:33.439575 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:33.439615 (+ 40us) server_negotiation.cc:407] Connection header received
0924 02:57:33.439735 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:33.439742 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:33.439795 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:33.439935 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:33.440500 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.441951 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.443000 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.443634 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.443762 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:33.446381 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:33.446385 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:33.446385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:33.446397 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:33.448393 (+ 1996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:33.448756 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:33.448758 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:33.448759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:33.448867 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:33.449033 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:33.449034 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:33.449034 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:33.449337 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:33.449495 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:33.449561 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 02:57:33.449697 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":74,"threads_started":1}
I20260924 02:57:33.451012 1010 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:57:33.451678 1010 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:33.457235 1010 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9032f57107a2474e9215bec4b7a4de29: Generated new TSK 0
I20260924 02:57:33.457401 1010 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 02:57:33.483464 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39689
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:33.561754 1024 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:33.561990 1024 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:33.562013 1024 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:33.563196 1024 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:33.563233 1024 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:33.563258 1024 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:57:33.564637 1024 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39689
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1024
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:33.564857 1024 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:33.565050 1024 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 02:57:33.569694 1024 server_base.cc:1034] running on GCE node
W20260924 02:57:33.569672 1030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:33.569680 1029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:33.569680 1032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:33.570003 1024 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:33.570228 1024 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:33.571355 1024 hybrid_clock.cc:648] HybridClock initialized: now 1790218653571343 us; error 30 us; skew 500 ppm
Sep 24 02:57:33 dist-test-slave-sb6p krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:33.573640 1024 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 02:57:33.574033 1024 webserver.cc:460] Webserver started at http://127.0.98.1:44729/ using document root <none> and password file <none>
I20260924 02:57:33.574250 1024 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:33.574297 1024 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:33.574414 1024 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:33.575450 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "e320eb1733c44b0fa55921ce0ab1ab37"
format_stamp: "Formatted at 2026-09-24 02:57:33 on dist-test-slave-sb6p"
server_key: "45df62b98c2128901f2e9814301b2f48"
server_key_iv: "71d668db05f9acd8442f229b8ea3b64c"
server_key_version: "encryptionkey@0"
I20260924 02:57:33.575781 1024 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e320eb1733c44b0fa55921ce0ab1ab37"
format_stamp: "Formatted at 2026-09-24 02:57:33 on dist-test-slave-sb6p"
server_key: "45df62b98c2128901f2e9814301b2f48"
server_key_iv: "71d668db05f9acd8442f229b8ea3b64c"
server_key_version: "encryptionkey@0"
I20260924 02:57:33.576925 1024 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:57:33.577589 1039 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:33.577736 1024 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:33.577811 1024 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "e320eb1733c44b0fa55921ce0ab1ab37"
format_stamp: "Formatted at 2026-09-24 02:57:33 on dist-test-slave-sb6p"
server_key: "45df62b98c2128901f2e9814301b2f48"
server_key_iv: "71d668db05f9acd8442f229b8ea3b64c"
server_key_version: "encryptionkey@0"
I20260924 02:57:33.577873 1024 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:33.612018 1024 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:33.614176 1024 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:33.614321 1024 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:33.614542 1024 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:33.614833 1024 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:33.614866 1024 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:33.614897 1024 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:33.614918 1024 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:33.620155 1024 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45721
I20260924 02:57:33.620203 1152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45721 every 8 connection(s)
I20260924 02:57:33.620492 1024 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 02:57:33 dist-test-slave-sb6p krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:33.627816 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 1024
I20260924 02:57:33.627910 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:57:33.628108 392 external_mini_cluster.cc:1428] Setting key 6ff54893a60b02ba3504b23e1a310562
I20260924 02:57:33.629084 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:33.621152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54951 (local address 127.0.98.62:39689)
0924 02:57:33.621206 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:33.621208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:33.621388 (+ 180us) server_negotiation.cc:407] Connection header received
0924 02:57:33.621891 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:33.621893 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:33.621904 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:33.621949 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:33.622883 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.623372 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.624310 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.624521 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.624573 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:33.626493 (+ 1920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:33.626494 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:33.626494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:33.626503 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:33.627744 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:33.628166 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:33.628167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:33.628167 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:33.628223 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:33.628461 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:33.628461 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:33.628461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:33.628568 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:33.628630 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:33.628915 (+ 285us) server_negotiation.cc:299] Negotiation successful
0924 02:57:33.628997 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 02:57:33.629243 1155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:33.621115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39689 (local address 127.0.98.1:54951)
0924 02:57:33.621273 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:33.621310 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:33.621800 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:33.622020 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:33.622023 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:33.622610 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:33.622798 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:33.622803 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.623447 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.623448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:33.624232 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:33.624235 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.624593 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.624594 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:33.624713 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:33.625137 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:33.625144 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:33.626415 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:33.627816 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:33.627823 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:33.627825 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:33.628086 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:33.628282 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:33.628282 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:33.628283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:33.628404 (+ 121us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:33.628629 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:33.628634 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:33.628789 (+ 155us) client_negotiation.cc:769] Sending connection context
0924 02:57:33.628886 (+ 97us) client_negotiation.cc:240] Negotiation successful
0924 02:57:33.629034 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
I20260924 02:57:33.629669 1153 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39689
I20260924 02:57:33.629768 1153 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:33.629952 1153 heartbeater.cc:511] Master 127.0.98.62:39689 requested a full tablet report, sending...
I20260924 02:57:33.630427 962 ts_manager.cc:194] Registered new tserver with Master: e320eb1733c44b0fa55921ce0ab1ab37 (127.0.98.1:45721)
I20260924 02:57:33.631016 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:54951
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 02:57:33.651340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39689
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:33.728250 1158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:33.728447 1158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:33.728477 1158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:33.729624 1158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:33.729666 1158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:33.729702 1158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:57:33.731099 1158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39689
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:33.731333 1158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:33.731527 1158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:33.736090 1163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:33.736116 1164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:33.736092 1166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:33.736164 1158 server_base.cc:1034] running on GCE node
I20260924 02:57:33.736521 1158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:33.736704 1158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:33.737834 1158 hybrid_clock.cc:648] HybridClock initialized: now 1790218653737817 us; error 29 us; skew 500 ppm
Sep 24 02:57:33 dist-test-slave-sb6p krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:33.740173 1158 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 02:57:33.740556 1158 webserver.cc:460] Webserver started at http://127.0.98.2:46821/ using document root <none> and password file <none>
I20260924 02:57:33.740747 1158 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:33.740789 1158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:33.740898 1158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:33.741881 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "23786b1579dc48b09d9106ca93e72023"
format_stamp: "Formatted at 2026-09-24 02:57:33 on dist-test-slave-sb6p"
server_key: "9bcf74f88d2e18b02e5eb9241f6c3eb7"
server_key_iv: "7229b3ca11f5b200fa54e37f002474b3"
server_key_version: "encryptionkey@0"
I20260924 02:57:33.742197 1158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "23786b1579dc48b09d9106ca93e72023"
format_stamp: "Formatted at 2026-09-24 02:57:33 on dist-test-slave-sb6p"
server_key: "9bcf74f88d2e18b02e5eb9241f6c3eb7"
server_key_iv: "7229b3ca11f5b200fa54e37f002474b3"
server_key_version: "encryptionkey@0"
I20260924 02:57:33.743376 1158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 02:57:33.743981 1173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:33.744124 1158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:33.744197 1158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "23786b1579dc48b09d9106ca93e72023"
format_stamp: "Formatted at 2026-09-24 02:57:33 on dist-test-slave-sb6p"
server_key: "9bcf74f88d2e18b02e5eb9241f6c3eb7"
server_key_iv: "7229b3ca11f5b200fa54e37f002474b3"
server_key_version: "encryptionkey@0"
I20260924 02:57:33.744256 1158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:33.774307 1158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:33.776150 1158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:33.776268 1158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:33.776481 1158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:33.776767 1158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:33.776799 1158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:33.776820 1158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:33.776844 1158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:33.782188 1158 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39647
I20260924 02:57:33.782241 1286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39647 every 8 connection(s)
I20260924 02:57:33.782552 1158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 02:57:33.785465 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 1158
I20260924 02:57:33.785540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:57:33.785689 392 external_mini_cluster.cc:1428] Setting key b1e55ed2a704329a0474930e3546149d
Sep 24 02:57:33 dist-test-slave-sb6p krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:33.792374 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:33.783237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55091 (local address 127.0.98.62:39689)
0924 02:57:33.783298 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:33.783300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:33.783488 (+ 188us) server_negotiation.cc:407] Connection header received
0924 02:57:33.783924 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:33.783925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:33.783935 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:33.783990 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:33.784853 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.785363 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.786596 (+ 1233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.786857 (+ 261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.786921 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:33.789185 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:33.789186 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:33.789186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:33.789194 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:33.790814 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:33.791239 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:33.791241 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:33.791242 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:33.791312 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:33.791582 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:33.791583 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:33.791584 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:33.791731 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:33.791790 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:33.792138 (+ 348us) server_negotiation.cc:299] Negotiation successful
0924 02:57:33.792239 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 02:57:33.792678 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:33.783209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39689 (local address 127.0.98.2:55091)
0924 02:57:33.783355 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:33.783395 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:33.783849 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:33.784050 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:33.784053 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:33.784592 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:33.784786 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:33.784790 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.785437 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.785438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:33.786513 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:33.786535 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.786926 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.786927 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:33.787023 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:33.787516 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:33.787526 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:33.789082 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:33.790878 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:33.790885 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:33.790887 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:33.791181 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:33.791382 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:33.791383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:33.791383 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:33.791513 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:33.791814 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:33.791819 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:33.791971 (+ 152us) client_negotiation.cc:769] Sending connection context
0924 02:57:33.792074 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 02:57:33.792233 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260924 02:57:33.793148 1287 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39689
I20260924 02:57:33.793253 1287 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:33.793445 1287 heartbeater.cc:511] Master 127.0.98.62:39689 requested a full tablet report, sending...
I20260924 02:57:33.793823 962 ts_manager.cc:194] Registered new tserver with Master: 23786b1579dc48b09d9106ca93e72023 (127.0.98.2:39647)
I20260924 02:57:33.794301 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55091
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 02:57:33.809056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39689
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36785
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:33.886451 1292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:33.886694 1292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:33.886726 1292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:33.887871 1292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:33.887912 1292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:33.887948 1292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:57:33.889379 1292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36785
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:39689
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:33.889613 1292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:33.889812 1292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:33.894331 1297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:33.894345 1292 server_base.cc:1034] running on GCE node
W20260924 02:57:33.894331 1300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:33.894344 1298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:33.894763 1292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:33.894950 1292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:33.896073 1292 hybrid_clock.cc:648] HybridClock initialized: now 1790218653896061 us; error 27 us; skew 500 ppm
Sep 24 02:57:33 dist-test-slave-sb6p krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:33.898244 1292 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 02:57:33.898665 1292 webserver.cc:460] Webserver started at http://127.0.98.3:40229/ using document root <none> and password file <none>
I20260924 02:57:33.898862 1292 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:33.898902 1292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:33.899020 1292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:33.900050 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "265b3ec065de4ee795a5030c717f025f"
format_stamp: "Formatted at 2026-09-24 02:57:33 on dist-test-slave-sb6p"
server_key: "242d3c5e0e765756b2a53772e1055453"
server_key_iv: "e14b8981c612d9617d9d754286a463b5"
server_key_version: "encryptionkey@0"
I20260924 02:57:33.900372 1292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "265b3ec065de4ee795a5030c717f025f"
format_stamp: "Formatted at 2026-09-24 02:57:33 on dist-test-slave-sb6p"
server_key: "242d3c5e0e765756b2a53772e1055453"
server_key_iv: "e14b8981c612d9617d9d754286a463b5"
server_key_version: "encryptionkey@0"
I20260924 02:57:33.901527 1292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 02:57:33.902143 1307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:33.902278 1292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:33.902350 1292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "265b3ec065de4ee795a5030c717f025f"
format_stamp: "Formatted at 2026-09-24 02:57:33 on dist-test-slave-sb6p"
server_key: "242d3c5e0e765756b2a53772e1055453"
server_key_iv: "e14b8981c612d9617d9d754286a463b5"
server_key_version: "encryptionkey@0"
I20260924 02:57:33.902413 1292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:33.916555 1292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:33.918119 1292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:33.918221 1292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:33.918380 1292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:33.918658 1292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:33.918699 1292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:33.918736 1292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:33.918752 1292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:33.923964 1292 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33327
I20260924 02:57:33.924000 1420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33327 every 8 connection(s)
I20260924 02:57:33.924283 1292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 02:57:33 dist-test-slave-sb6p krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:33.931974 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:33.924870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57983 (local address 127.0.98.62:39689)
0924 02:57:33.924928 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:33.924929 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:33.925103 (+ 174us) server_negotiation.cc:407] Connection header received
0924 02:57:33.925557 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:33.925558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:33.925567 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:33.925611 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:33.926481 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.927005 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.927911 (+ 906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.928078 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.928123 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:33.929919 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:33.929920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:33.929920 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:33.929926 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:33.931010 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:33.931324 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:33.931325 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:33.931325 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:33.931366 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:33.931526 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:33.931527 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:33.931527 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:33.931626 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:33.931671 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:33.931860 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 02:57:33.931913 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 02:57:33.932138 1423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:33.924848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:39689 (local address 127.0.98.3:57983)
0924 02:57:33.924986 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:33.925024 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:33.925470 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:33.925675 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:33.925678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:33.926206 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:33.926409 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:33.926414 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.927075 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.927076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:33.927851 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:33.927853 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.928130 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.928131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:33.928226 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:33.928601 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:33.928621 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:33.929842 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:33.931077 (+ 1235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:33.931082 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:33.931084 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:33.931274 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:33.931407 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:33.931408 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:33.931408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:33.931488 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:33.931667 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:33.931670 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:33.931771 (+ 101us) client_negotiation.cc:769] Sending connection context
0924 02:57:33.931843 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 02:57:33.931948 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":77,"threads_started":1}
I20260924 02:57:33.932524 1421 heartbeater.cc:347] Connected to a master server at 127.0.98.62:39689
I20260924 02:57:33.932600 1421 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:33.932756 1421 heartbeater.cc:511] Master 127.0.98.62:39689 requested a full tablet report, sending...
I20260924 02:57:33.932981 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 1292
I20260924 02:57:33.933074 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:57:33.933099 962 ts_manager.cc:194] Registered new tserver with Master: 265b3ec065de4ee795a5030c717f025f (127.0.98.3:33327)
I20260924 02:57:33.933246 392 external_mini_cluster.cc:1428] Setting key 0e071674245c7d7c988f1d58cb2f7e79
I20260924 02:57:33.933530 962 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57983
I20260924 02:57:33.934054 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 02:57:33.940872 1020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:33.935435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38154 (local address 127.0.98.62:39689)
0924 02:57:33.935471 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:33.935473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:33.935569 (+ 96us) server_negotiation.cc:407] Connection header received
0924 02:57:33.935630 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:33.935631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:33.935640 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:33.935679 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:33.936268 (+ 589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.936873 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.937569 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.937832 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.937885 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:33.938947 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:33.938948 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:33.938949 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:33.938955 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:33.939925 (+ 970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:33.940204 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:33.940205 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:33.940205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:33.940254 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:33.940405 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:33.940406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:33.940406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:33.940527 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:33.940608 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:33.940727 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 02:57:33.940791 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 02:57:33.942665 962 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 02:57:33.944137 962 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:57:33.949864 1429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:33.945899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45721 (local address 127.0.0.1:52548)
0924 02:57:33.946025 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:33.946063 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:33.946129 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:33.946407 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:33.946410 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:33.946412 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:33.946475 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:33.946477 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.947913 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.947914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:33.948729 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:33.948731 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.949584 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.949586 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:33.949671 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:33.949685 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 02:57:33.949745 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 02:57:33.949797 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":63,"threads_started":1}
I20260924 02:57:33.950058 1430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:33.946188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33327 (local address 127.0.0.1:54902)
0924 02:57:33.946379 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:33.946389 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:33.946437 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:33.946606 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:33.946607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:33.946609 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:33.946690 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:33.946692 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.947887 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.947888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:33.949037 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:33.949039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.949844 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.949845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:33.949903 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:33.949907 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:57:33.949954 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 02:57:33.949994 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":29,"thread_start_us":47,"threads_started":1}
I20260924 02:57:33.950160 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:33.946019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52548 (local address 127.0.98.1:45721)
0924 02:57:33.946238 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:33.946243 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:33.946252 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:57:33.946288 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:33.946292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:33.946307 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:33.946568 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:33.946640 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.947809 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.948806 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.949507 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.949576 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:33.949630 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:33.950035 (+ 405us) server_negotiation.cc:299] Negotiation successful
0924 02:57:33.950072 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":117,"threads_started":1}
I20260924 02:57:33.950181 1432 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:33.946078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39647 (local address 127.0.0.1:52172)
0924 02:57:33.946273 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:33.946281 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:33.946317 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:33.946707 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:33.946708 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:33.946709 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:33.946737 (+ 28us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:33.946738 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.947783 (+ 1045us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.947784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:33.949101 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:33.949102 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.949989 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.949990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:33.950047 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:33.950050 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:33.950093 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 02:57:33.950127 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":7,"thread_start_us":35,"threads_started":1}
I20260924 02:57:33.950323 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:33.946316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54902 (local address 127.0.98.3:33327)
0924 02:57:33.946450 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:33.946455 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:33.946464 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:57:33.946495 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:33.946497 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:33.946511 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:33.946616 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:33.946745 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.947786 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.949108 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.949775 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.949840 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:33.949887 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:33.950208 (+ 321us) server_negotiation.cc:299] Negotiation successful
0924 02:57:33.950252 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260924 02:57:33.950788 1087 tablet_service.cc:1467] Processing CreateTablet for tablet e6079be76b164998badcfed0c450168b (DEFAULT_TABLE table=test-table [id=905bbdef04c6448d98b2a766be98c764]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:33.950888 1355 tablet_service.cc:1467] Processing CreateTablet for tablet e6079be76b164998badcfed0c450168b (DEFAULT_TABLE table=test-table [id=905bbdef04c6448d98b2a766be98c764]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:33.951143 1087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6079be76b164998badcfed0c450168b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:33.951174 1355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6079be76b164998badcfed0c450168b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:33.954241 1435 tablet_bootstrap.cc:492] T e6079be76b164998badcfed0c450168b P e320eb1733c44b0fa55921ce0ab1ab37: Bootstrap starting.
I20260924 02:57:33.954316 1433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:33.946194 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52172 (local address 127.0.98.2:39647)
0924 02:57:33.946505 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:33.946509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:33.946545 (+ 36us) server_negotiation.cc:407] Connection header received
0924 02:57:33.946597 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:33.946599 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:33.946614 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:33.946798 (+ 184us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:33.946853 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.947672 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.949155 (+ 1483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:33.949671 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:33.949895 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:33.949961 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:33.950307 (+ 346us) server_negotiation.cc:299] Negotiation successful
0924 02:57:33.950350 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":100,"threads_started":1}
I20260924 02:57:33.954324 1436 tablet_bootstrap.cc:492] T e6079be76b164998badcfed0c450168b P 265b3ec065de4ee795a5030c717f025f: Bootstrap starting.
I20260924 02:57:33.954869 1436 tablet_bootstrap.cc:654] T e6079be76b164998badcfed0c450168b P 265b3ec065de4ee795a5030c717f025f: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:33.954914 1435 tablet_bootstrap.cc:654] T e6079be76b164998badcfed0c450168b P e320eb1733c44b0fa55921ce0ab1ab37: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:33.954955 1221 tablet_service.cc:1467] Processing CreateTablet for tablet e6079be76b164998badcfed0c450168b (DEFAULT_TABLE table=test-table [id=905bbdef04c6448d98b2a766be98c764]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:33.955304 1221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6079be76b164998badcfed0c450168b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:33.955359 1436 log.cc:824] T e6079be76b164998badcfed0c450168b P 265b3ec065de4ee795a5030c717f025f: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:33.955452 1435 log.cc:824] T e6079be76b164998badcfed0c450168b P e320eb1733c44b0fa55921ce0ab1ab37: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:33.956007 1436 tablet_bootstrap.cc:492] T e6079be76b164998badcfed0c450168b P 265b3ec065de4ee795a5030c717f025f: No bootstrap required, opened a new log
I20260924 02:57:33.956066 1436 ts_tablet_manager.cc:1397] T e6079be76b164998badcfed0c450168b P 265b3ec065de4ee795a5030c717f025f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 02:57:33.956137 1435 tablet_bootstrap.cc:492] T e6079be76b164998badcfed0c450168b P e320eb1733c44b0fa55921ce0ab1ab37: No bootstrap required, opened a new log
I20260924 02:57:33.956204 1435 ts_tablet_manager.cc:1397] T e6079be76b164998badcfed0c450168b P e320eb1733c44b0fa55921ce0ab1ab37: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 02:57:33.957389 1436 raft_consensus.cc:348] T e6079be76b164998badcfed0c450168b P 265b3ec065de4ee795a5030c717f025f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e320eb1733c44b0fa55921ce0ab1ab37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "23786b1579dc48b09d9106ca93e72023" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39647 } } peers { permanent_uuid: "265b3ec065de4ee795a5030c717f025f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33327 } }
I20260924 02:57:33.957517 1436 raft_consensus.cc:374] T e6079be76b164998badcfed0c450168b P 265b3ec065de4ee795a5030c717f025f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:33.957557 1436 raft_consensus.cc:729] T e6079be76b164998badcfed0c450168b P 265b3ec065de4ee795a5030c717f025f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265b3ec065de4ee795a5030c717f025f, State: Initialized, Role: FOLLOWER
I20260924 02:57:33.957677 1436 consensus_queue.cc:260] T e6079be76b164998badcfed0c450168b P 265b3ec065de4ee795a5030c717f025f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e320eb1733c44b0fa55921ce0ab1ab37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "23786b1579dc48b09d9106ca93e72023" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39647 } } peers { permanent_uuid: "265b3ec065de4ee795a5030c717f025f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33327 } }
I20260924 02:57:33.957947 1436 ts_tablet_manager.cc:1428] T e6079be76b164998badcfed0c450168b P 265b3ec065de4ee795a5030c717f025f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 02:57:33.957964 1421 heartbeater.cc:503] Master 127.0.98.62:39689 was elected leader, sending a full tablet report...
I20260924 02:57:33.958132 1435 raft_consensus.cc:348] T e6079be76b164998badcfed0c450168b P e320eb1733c44b0fa55921ce0ab1ab37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e320eb1733c44b0fa55921ce0ab1ab37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "23786b1579dc48b09d9106ca93e72023" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39647 } } peers { permanent_uuid: "265b3ec065de4ee795a5030c717f025f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33327 } }
I20260924 02:57:33.958298 1435 raft_consensus.cc:374] T e6079be76b164998badcfed0c450168b P e320eb1733c44b0fa55921ce0ab1ab37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:33.958340 1435 raft_consensus.cc:729] T e6079be76b164998badcfed0c450168b P e320eb1733c44b0fa55921ce0ab1ab37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e320eb1733c44b0fa55921ce0ab1ab37, State: Initialized, Role: FOLLOWER
I20260924 02:57:33.958444 1435 consensus_queue.cc:260] T e6079be76b164998badcfed0c450168b P e320eb1733c44b0fa55921ce0ab1ab37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e320eb1733c44b0fa55921ce0ab1ab37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "23786b1579dc48b09d9106ca93e72023" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39647 } } peers { permanent_uuid: "265b3ec065de4ee795a5030c717f025f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33327 } }
I20260924 02:57:33.958724 1153 heartbeater.cc:503] Master 127.0.98.62:39689 was elected leader, sending a full tablet report...
I20260924 02:57:33.959017 1435 ts_tablet_manager.cc:1428] T e6079be76b164998badcfed0c450168b P e320eb1733c44b0fa55921ce0ab1ab37: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260924 02:57:33.959211 1441 tablet_bootstrap.cc:492] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023: Bootstrap starting.
I20260924 02:57:33.959782 1441 tablet_bootstrap.cc:654] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:33.960175 1441 log.cc:824] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:33.960714 1441 tablet_bootstrap.cc:492] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023: No bootstrap required, opened a new log
I20260924 02:57:33.960764 1441 ts_tablet_manager.cc:1397] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:33.961962 1441 raft_consensus.cc:348] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e320eb1733c44b0fa55921ce0ab1ab37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "23786b1579dc48b09d9106ca93e72023" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39647 } } peers { permanent_uuid: "265b3ec065de4ee795a5030c717f025f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33327 } }
I20260924 02:57:33.962085 1441 raft_consensus.cc:374] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:33.962114 1441 raft_consensus.cc:729] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23786b1579dc48b09d9106ca93e72023, State: Initialized, Role: FOLLOWER
I20260924 02:57:33.962229 1441 consensus_queue.cc:260] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e320eb1733c44b0fa55921ce0ab1ab37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "23786b1579dc48b09d9106ca93e72023" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39647 } } peers { permanent_uuid: "265b3ec065de4ee795a5030c717f025f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33327 } }
I20260924 02:57:33.962471 1287 heartbeater.cc:503] Master 127.0.98.62:39689 was elected leader, sending a full tablet report...
I20260924 02:57:33.962576 1441 ts_tablet_manager.cc:1428] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260924 02:57:34.033082 1288 tablet.cc:2369] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 02:57:34.121156 1154 tablet.cc:2369] T e6079be76b164998badcfed0c450168b P e320eb1733c44b0fa55921ce0ab1ab37: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 02:57:34.174796 1422 tablet.cc:2369] T e6079be76b164998badcfed0c450168b P 265b3ec065de4ee795a5030c717f025f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 02:57:34.236951 1443 raft_consensus.cc:482] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:57:34.237090 1443 raft_consensus.cc:504] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e320eb1733c44b0fa55921ce0ab1ab37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "23786b1579dc48b09d9106ca93e72023" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39647 } } peers { permanent_uuid: "265b3ec065de4ee795a5030c717f025f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33327 } }
I20260924 02:57:34.237423 1443 leader_election.cc:288] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e320eb1733c44b0fa55921ce0ab1ab37 (127.0.98.1:45721), 265b3ec065de4ee795a5030c717f025f (127.0.98.3:33327)
I20260924 02:57:34.240341 1289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:34.237603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45721 (local address 127.0.98.2:35699)
0924 02:57:34.237671 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:34.237682 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:34.237742 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:34.237932 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:34.237932 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:34.237934 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:34.238000 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:34.238001 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:34.238729 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:34.238731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:34.239550 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:34.239552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:34.240087 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:34.240088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:34.240190 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:34.240194 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:57:34.240242 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 02:57:34.240273 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49,"mutex_wait_us":31}
I20260924 02:57:34.240356 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:34.237677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35699 (local address 127.0.98.1:45721)
0924 02:57:34.237727 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:34.237729 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:34.237738 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:57:34.237834 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:34.237835 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:34.237845 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:34.237887 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:34.238070 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:34.238663 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:34.239631 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:34.240020 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:34.240070 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:34.240079 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:34.240253 (+ 174us) server_negotiation.cc:299] Negotiation successful
0924 02:57:34.240288 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 02:57:34.240433 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:34.237841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37919 (local address 127.0.98.3:33327)
0924 02:57:34.237885 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:34.237887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:34.237972 (+ 85us) server_negotiation.cc:407] Connection header received
0924 02:57:34.238023 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:34.238024 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:34.238034 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:34.238085 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:34.238273 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:34.238789 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:34.239747 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:34.240126 (+ 379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:34.240190 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:34.240199 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:34.240337 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 02:57:34.240378 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 02:57:34.240626 1444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:34.237723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:33327 (local address 127.0.98.2:37919)
0924 02:57:34.237910 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:34.237918 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:34.237973 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:34.238092 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:34.238093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:34.238095 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:34.238160 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:34.238162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:34.238878 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:34.238879 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:34.239669 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:34.239671 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:34.240202 (+ 531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:34.240203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:34.240265 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:34.240269 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:57:34.240319 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 02:57:34.240353 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":104,"threads_started":1}
I20260924 02:57:34.240680 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6079be76b164998badcfed0c450168b" candidate_uuid: "23786b1579dc48b09d9106ca93e72023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e320eb1733c44b0fa55921ce0ab1ab37" is_pre_election: true
I20260924 02:57:34.240835 1107 raft_consensus.cc:2454] T e6079be76b164998badcfed0c450168b P e320eb1733c44b0fa55921ce0ab1ab37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23786b1579dc48b09d9106ca93e72023 in term 0.
I20260924 02:57:34.241065 1175 leader_election.cc:302] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23786b1579dc48b09d9106ca93e72023, e320eb1733c44b0fa55921ce0ab1ab37; no voters:
I20260924 02:57:34.241196 1443 raft_consensus.cc:2781] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:57:34.241187 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6079be76b164998badcfed0c450168b" candidate_uuid: "23786b1579dc48b09d9106ca93e72023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265b3ec065de4ee795a5030c717f025f" is_pre_election: true
I20260924 02:57:34.241248 1443 raft_consensus.cc:482] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:57:34.241263 1443 raft_consensus.cc:3037] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:34.241317 1375 raft_consensus.cc:2454] T e6079be76b164998badcfed0c450168b P 265b3ec065de4ee795a5030c717f025f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23786b1579dc48b09d9106ca93e72023 in term 0.
I20260924 02:57:34.241981 1443 raft_consensus.cc:504] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e320eb1733c44b0fa55921ce0ab1ab37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "23786b1579dc48b09d9106ca93e72023" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39647 } } peers { permanent_uuid: "265b3ec065de4ee795a5030c717f025f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33327 } }
I20260924 02:57:34.242084 1443 leader_election.cc:288] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 [CANDIDATE]: Term 1 election: Requested vote from peers e320eb1733c44b0fa55921ce0ab1ab37 (127.0.98.1:45721), 265b3ec065de4ee795a5030c717f025f (127.0.98.3:33327)
I20260924 02:57:34.242235 1107 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6079be76b164998badcfed0c450168b" candidate_uuid: "23786b1579dc48b09d9106ca93e72023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e320eb1733c44b0fa55921ce0ab1ab37"
I20260924 02:57:34.242249 1375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6079be76b164998badcfed0c450168b" candidate_uuid: "23786b1579dc48b09d9106ca93e72023" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "265b3ec065de4ee795a5030c717f025f"
I20260924 02:57:34.242300 1107 raft_consensus.cc:3037] T e6079be76b164998badcfed0c450168b P e320eb1733c44b0fa55921ce0ab1ab37 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:34.242314 1375 raft_consensus.cc:3037] T e6079be76b164998badcfed0c450168b P 265b3ec065de4ee795a5030c717f025f [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:34.242977 1107 raft_consensus.cc:2454] T e6079be76b164998badcfed0c450168b P e320eb1733c44b0fa55921ce0ab1ab37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23786b1579dc48b09d9106ca93e72023 in term 1.
I20260924 02:57:34.243175 1175 leader_election.cc:302] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23786b1579dc48b09d9106ca93e72023, e320eb1733c44b0fa55921ce0ab1ab37; no voters:
I20260924 02:57:34.243254 1443 raft_consensus.cc:2781] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:34.243414 1443 raft_consensus.cc:686] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 [term 1 LEADER]: Becoming Leader. State: Replica: 23786b1579dc48b09d9106ca93e72023, State: Running, Role: LEADER
I20260924 02:57:34.243510 1443 consensus_queue.cc:237] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e320eb1733c44b0fa55921ce0ab1ab37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } } peers { permanent_uuid: "23786b1579dc48b09d9106ca93e72023" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39647 } } peers { permanent_uuid: "265b3ec065de4ee795a5030c717f025f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33327 } }
I20260924 02:57:34.243569 1375 raft_consensus.cc:2454] T e6079be76b164998badcfed0c450168b P 265b3ec065de4ee795a5030c717f025f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23786b1579dc48b09d9106ca93e72023 in term 1.
I20260924 02:57:34.244342 962 catalog_manager.cc:5495] T e6079be76b164998badcfed0c450168b P 23786b1579dc48b09d9106ca93e72023 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23786b1579dc48b09d9106ca93e72023 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "23786b1579dc48b09d9106ca93e72023" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e320eb1733c44b0fa55921ce0ab1ab37" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45721 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23786b1579dc48b09d9106ca93e72023" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39647 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "265b3ec065de4ee795a5030c717f025f" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 33327 } health_report { overall_health: UNKNOWN } } }
Sep 24 02:57:34 dist-test-slave-sb6p krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 02:57:34.304651 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:34.298183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52564 (local address 127.0.98.1:45721)
0924 02:57:34.298240 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:34.298242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:34.298278 (+ 36us) server_negotiation.cc:407] Connection header received
0924 02:57:34.298340 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:34.298341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:34.298350 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:34.298394 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:34.298892 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:34.299352 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:34.300090 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:34.300290 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:34.300355 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:34.302383 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:34.302395 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:34.302396 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:34.302402 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:34.303748 (+ 1346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:34.304025 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:34.304029 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:34.304029 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:34.304114 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:34.304238 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:34.304238 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:34.304238 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:34.304357 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:34.304448 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:34.304498 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 02:57:34.304571 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 02:57:34 dist-test-slave-sb6p krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218654, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 02:57:34 dist-test-slave-sb6p krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218654, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 02:57:34.325551 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:34.319345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52570 (local address 127.0.98.1:45721)
0924 02:57:34.319402 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:34.319404 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:34.319439 (+ 35us) server_negotiation.cc:407] Connection header received
0924 02:57:34.319522 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:34.319522 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:34.319531 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:34.319575 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:34.320083 (+ 508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:34.320576 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:34.321281 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:34.321473 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:34.321524 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:34.323519 (+ 1995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:34.323520 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:34.323521 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:34.323528 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:34.324557 (+ 1029us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:34.324850 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:34.324852 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:34.324852 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:34.324900 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:34.325052 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:34.325053 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:34.325054 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:34.325147 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:34.325226 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:34.325388 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 02:57:34.325461 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 02:57:34 dist-test-slave-sb6p krb5kdc[917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 02:57:34 dist-test-slave-sb6p krb5kdc[917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218654, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 02:57:34.346035 1431 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:34.339958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52578 (local address 127.0.98.1:45721)
0924 02:57:34.340021 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:34.340023 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:34.340113 (+ 90us) server_negotiation.cc:407] Connection header received
0924 02:57:34.340179 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:34.340180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:34.340190 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:34.340233 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:34.340722 (+ 489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:34.341187 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:34.341878 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:34.342054 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:34.342112 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:34.344047 (+ 1935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:34.344049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:34.344049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:34.344056 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:34.345063 (+ 1007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:34.345373 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:34.345375 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:34.345375 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:34.345422 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:34.345589 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:34.345590 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:34.345590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:34.345681 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:34.345762 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:34.345855 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 02:57:34.345926 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260924 02:57:34.346446 1067 tablet_service.cc:2908] Rejecting scan request for tablet e6079be76b164998badcfed0c450168b: Uninitialized: clean time has not yet been initialized
I20260924 02:57:34.347357 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 1024
I20260924 02:57:34.351837 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 1158
I20260924 02:57:34.355598 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 1292
I20260924 02:57:34.359206 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 930
2026-09-24T02:57:34Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2104 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:57:34 dist-test-slave-sb6p krb5kdc[1459](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:57:34 dist-test-slave-sb6p krb5kdc[1459](info): set up 2 sockets
Sep 24 02:57:34 dist-test-slave-sb6p krb5kdc[1459](info): commencing operation
krb5kdc: starting...
W20260924 02:57:35.313892 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:57:35 dist-test-slave-sb6p krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:57:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:57:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 02:57:35.421271 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43965
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43965
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:35.498392 1472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:35.498622 1472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:35.498654 1472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:35.499810 1472 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:35.499848 1472 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 02:57:35.499866 1472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:35.499882 1472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:35.499897 1472 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:35.499912 1472 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:57:35.501425 1472 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43965
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1472
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:35.501665 1472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:35.501863 1472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:35.506624 1478 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:35.506615 1480 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:35.506754 1472 server_base.cc:1034] running on GCE node
W20260924 02:57:35.506616 1477 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:35.507098 1472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:35.507345 1472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:35.508486 1472 hybrid_clock.cc:648] HybridClock initialized: now 1790218655508469 us; error 28 us; skew 500 ppm
Sep 24 02:57:35 dist-test-slave-sb6p krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:35.510782 1472 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:57:35.511160 1472 webserver.cc:460] Webserver started at http://127.0.98.62:39377/ using document root <none> and password file <none>
I20260924 02:57:35.511358 1472 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:35.511400 1472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:35.511519 1472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:35.512522 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "9f9ea3e78e06423080db8773cb4efcb8"
format_stamp: "Formatted at 2026-09-24 02:57:35 on dist-test-slave-sb6p"
server_key: "e4bb1be0a680c4b3190d6ad900530b76"
server_key_iv: "f447fb9f9fcc677fc6a23605da69bd4f"
server_key_version: "encryptionkey@0"
I20260924 02:57:35.512840 1472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "9f9ea3e78e06423080db8773cb4efcb8"
format_stamp: "Formatted at 2026-09-24 02:57:35 on dist-test-slave-sb6p"
server_key: "e4bb1be0a680c4b3190d6ad900530b76"
server_key_iv: "f447fb9f9fcc677fc6a23605da69bd4f"
server_key_version: "encryptionkey@0"
I20260924 02:57:35.514008 1472 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 02:57:35.514665 1487 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:35.514801 1472 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:35.514894 1472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "9f9ea3e78e06423080db8773cb4efcb8"
format_stamp: "Formatted at 2026-09-24 02:57:35 on dist-test-slave-sb6p"
server_key: "e4bb1be0a680c4b3190d6ad900530b76"
server_key_iv: "f447fb9f9fcc677fc6a23605da69bd4f"
server_key_version: "encryptionkey@0"
I20260924 02:57:35.514955 1472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:35.526635 1472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:35.528647 1472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:35.528771 1472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:35.532891 1472 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43965
I20260924 02:57:35.532949 1549 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43965 every 8 connection(s)
I20260924 02:57:35.533241 1472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:57:35.533840 1550 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:35.535312 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 1472
I20260924 02:57:35.535403 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:57:35.535553 392 external_mini_cluster.cc:1428] Setting key ce9131ca8caaee99332740f32a79215c
I20260924 02:57:35.536309 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8: Bootstrap starting.
I20260924 02:57:35.536850 1550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:35.537115 1550 log.cc:824] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:35.537709 1550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8: No bootstrap required, opened a new log
I20260924 02:57:35.539450 1550 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ea3e78e06423080db8773cb4efcb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43965 } }
I20260924 02:57:35.539588 1550 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:35.539620 1550 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f9ea3e78e06423080db8773cb4efcb8, State: Initialized, Role: FOLLOWER
I20260924 02:57:35.539754 1550 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ea3e78e06423080db8773cb4efcb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43965 } }
I20260924 02:57:35.539840 1550 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:57:35.539876 1550 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:57:35.539919 1550 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:35.540417 1550 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ea3e78e06423080db8773cb4efcb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43965 } }
I20260924 02:57:35.540503 1550 leader_election.cc:302] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9f9ea3e78e06423080db8773cb4efcb8; no voters:
I20260924 02:57:35.540650 1550 leader_election.cc:288] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:57:35.540679 1555 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:35.540792 1555 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [term 1 LEADER]: Becoming Leader. State: Replica: 9f9ea3e78e06423080db8773cb4efcb8, State: Running, Role: LEADER
I20260924 02:57:35.540871 1555 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ea3e78e06423080db8773cb4efcb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43965 } }
I20260924 02:57:35.540898 1550 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:35.541181 1555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9f9ea3e78e06423080db8773cb4efcb8. Latest consensus state: current_term: 1 leader_uuid: "9f9ea3e78e06423080db8773cb4efcb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ea3e78e06423080db8773cb4efcb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43965 } } }
I20260924 02:57:35.541260 1555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:35.541193 1556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9f9ea3e78e06423080db8773cb4efcb8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9f9ea3e78e06423080db8773cb4efcb8" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43965 } } }
I20260924 02:57:35.541422 1556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8 [sys.catalog]: This master's current role is: LEADER
Sep 24 02:57:35 dist-test-slave-sb6p krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218655, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:35.542057 1560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:35.542690 1560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:57:35.544844 1560 catalog_manager.cc:1347] Generated new cluster ID: cd030ea7db9f4d8795fc4a640f142847
I20260924 02:57:35.544889 1560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:57:35.545231 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:35.536044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59750 (local address 127.0.98.62:43965)
0924 02:57:35.536205 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:35.536214 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:35.536237 (+ 23us) server_negotiation.cc:407] Connection header received
0924 02:57:35.536659 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:35.536665 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:35.536713 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:35.536837 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:35.537452 (+ 615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.538349 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.539099 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.539495 (+ 396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.539588 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:35.541913 (+ 2325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:35.541917 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:35.541918 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:35.541930 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:35.543622 (+ 1692us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:35.544018 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:35.544021 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:35.544021 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:35.544133 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:35.544356 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:35.544357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:35.544357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:35.544656 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:35.544786 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:35.544905 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 02:57:35.545035 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":74,"threads_started":1}
I20260924 02:57:35.568957 1560 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:57:35.569701 1560 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:35.578689 1560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8: Generated new TSK 0
I20260924 02:57:35.578872 1560 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 02:57:35.608664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43965
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:35.694602 1576 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:35.694797 1576 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:35.694819 1576 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:35.695982 1576 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:35.696023 1576 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:35.696043 1576 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 02:57:35.696061 1576 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:57:35.697531 1576 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1576
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:35.697743 1576 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:35.697952 1576 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:35.702497 1581 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:35.702543 1582 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:35.702749 1584 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:35.702682 1576 server_base.cc:1034] running on GCE node
I20260924 02:57:35.702937 1576 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:35.703190 1576 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:35.704350 1576 hybrid_clock.cc:648] HybridClock initialized: now 1790218655704335 us; error 30 us; skew 500 ppm
Sep 24 02:57:35 dist-test-slave-sb6p krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:35.706856 1576 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 02:57:35.707263 1576 webserver.cc:460] Webserver started at http://127.0.98.1:34047/ using document root <none> and password file <none>
I20260924 02:57:35.707463 1576 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:35.707508 1576 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:35.707626 1576 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:35.708612 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "f632640b969c4ac4954fb8c033e101f6"
format_stamp: "Formatted at 2026-09-24 02:57:35 on dist-test-slave-sb6p"
server_key: "9bd5d85cd49dd51a3e1f7f21cfb9732c"
server_key_iv: "6b3b1b96c5430d140f3486a2c8029449"
server_key_version: "encryptionkey@0"
I20260924 02:57:35.708925 1576 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f632640b969c4ac4954fb8c033e101f6"
format_stamp: "Formatted at 2026-09-24 02:57:35 on dist-test-slave-sb6p"
server_key: "9bd5d85cd49dd51a3e1f7f21cfb9732c"
server_key_iv: "6b3b1b96c5430d140f3486a2c8029449"
server_key_version: "encryptionkey@0"
I20260924 02:57:35.710083 1576 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 02:57:35.710711 1591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:35.710834 1576 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:35.710911 1576 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "f632640b969c4ac4954fb8c033e101f6"
format_stamp: "Formatted at 2026-09-24 02:57:35 on dist-test-slave-sb6p"
server_key: "9bd5d85cd49dd51a3e1f7f21cfb9732c"
server_key_iv: "6b3b1b96c5430d140f3486a2c8029449"
server_key_version: "encryptionkey@0"
I20260924 02:57:35.710971 1576 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:35.723552 1576 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:35.725591 1576 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:35.725723 1576 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:35.726229 1576 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:35.726269 1576 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:35.726296 1576 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:35.726317 1576 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 02:57:35 dist-test-slave-sb6p krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:35.732825 1576 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40357
I20260924 02:57:35.733152 1576 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:57:35.733170 1706 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40357 every 8 connection(s)
I20260924 02:57:35.734055 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 1576
I20260924 02:57:35.734148 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:57:35.734344 392 external_mini_cluster.cc:1428] Setting key b1fff276feb7ff301435550be5935906
I20260924 02:57:35.735250 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:35.726666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40577 (local address 127.0.98.62:43965)
0924 02:57:35.726714 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:35.726716 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:35.726926 (+ 210us) server_negotiation.cc:407] Connection header received
0924 02:57:35.727411 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:35.727412 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:35.727425 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:35.727502 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:35.728274 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.728836 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.729819 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.730030 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.730087 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:35.732492 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:35.732493 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:35.732494 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:35.732510 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:35.733813 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:35.734262 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:35.734263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:35.734264 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:35.734321 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:35.734599 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:35.734600 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:35.734600 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:35.734719 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:35.734778 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:35.735075 (+ 297us) server_negotiation.cc:299] Negotiation successful
0924 02:57:35.735164 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 02:57:35.735397 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:35.726685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43965 (local address 127.0.98.1:40577)
0924 02:57:35.726816 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:35.726849 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:35.727317 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:35.727627 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:35.727630 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:35.727901 (+ 271us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:35.728162 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:35.728168 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.728921 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.728922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:35.729741 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:35.729743 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.730098 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.730099 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:35.730216 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:35.730694 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:35.730702 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:35.732409 (+ 1707us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:35.733882 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:35.733888 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:35.733891 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:35.734177 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:35.734388 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:35.734389 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:35.734389 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:35.734542 (+ 153us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:35.734785 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:35.734792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:35.734942 (+ 150us) client_negotiation.cc:769] Sending connection context
0924 02:57:35.735045 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 02:57:35.735197 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":72,"threads_started":1}
I20260924 02:57:35.736660 1707 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43965
I20260924 02:57:35.736740 1707 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:35.736903 1707 heartbeater.cc:511] Master 127.0.98.62:43965 requested a full tablet report, sending...
I20260924 02:57:35.737448 1503 ts_manager.cc:194] Registered new tserver with Master: f632640b969c4ac4954fb8c033e101f6 (127.0.98.1:40357)
I20260924 02:57:35.738305 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:40577
I20260924 02:57:35.742481 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:35.736923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58351 (local address 127.0.98.62:43965)
0924 02:57:35.736969 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:35.736971 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:35.736977 (+ 6us) server_negotiation.cc:407] Connection header received
0924 02:57:35.736994 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:35.736994 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:35.737002 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:35.737046 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:35.737368 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.737833 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.738649 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.738925 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.738998 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:35.739830 (+ 832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:35.739831 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:35.739832 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:35.739838 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:35.741286 (+ 1448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:35.741650 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:35.741651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:35.741652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:35.741702 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:35.741923 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:35.741924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:35.741924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:35.742119 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:35.742180 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:35.742363 (+ 183us) server_negotiation.cc:299] Negotiation successful
0924 02:57:35.742423 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 02:57:35.742812 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:35.736752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43965 (local address 127.0.98.1:58351)
0924 02:57:35.736864 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:35.736874 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:35.736926 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:35.737062 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:35.737062 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:35.737246 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:35.737306 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:35.737308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.737913 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.737914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:35.738550 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:35.738552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.739005 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.739006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:35.739065 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:35.739475 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:35.739479 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:35.739749 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:35.741376 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:35.741377 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:35.741377 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:35.741584 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:35.741765 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:35.741765 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:35.741766 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:35.741828 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:35.742190 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:35.742191 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:35.742246 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 02:57:35.742324 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 02:57:35.742420 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":61,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 02:57:35.758666 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43965
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:35.839468 1716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:35.839668 1716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:35.839690 1716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:35.840788 1716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:35.840821 1716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:35.840839 1716 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 02:57:35.840855 1716 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:57:35.842355 1716 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:35.842597 1716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:35.842783 1716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:35.847450 1722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:35.847450 1724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:35.847736 1721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:35.847707 1716 server_base.cc:1034] running on GCE node
I20260924 02:57:35.847946 1716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:35.848218 1716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:35.849375 1716 hybrid_clock.cc:648] HybridClock initialized: now 1790218655849364 us; error 30 us; skew 500 ppm
Sep 24 02:57:35 dist-test-slave-sb6p krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:35.851855 1716 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 02:57:35.852288 1716 webserver.cc:460] Webserver started at http://127.0.98.2:33185/ using document root <none> and password file <none>
I20260924 02:57:35.852499 1716 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:35.852547 1716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:35.852669 1716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:35.853673 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "79585bebb1c94b6ea469ae48b52655bc"
format_stamp: "Formatted at 2026-09-24 02:57:35 on dist-test-slave-sb6p"
server_key: "15c05693dd42da9f27a6855f7797cbb2"
server_key_iv: "376f7a2a9358e7b23d5a676261f1712a"
server_key_version: "encryptionkey@0"
I20260924 02:57:35.854132 1716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "79585bebb1c94b6ea469ae48b52655bc"
format_stamp: "Formatted at 2026-09-24 02:57:35 on dist-test-slave-sb6p"
server_key: "15c05693dd42da9f27a6855f7797cbb2"
server_key_iv: "376f7a2a9358e7b23d5a676261f1712a"
server_key_version: "encryptionkey@0"
I20260924 02:57:35.855530 1716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 02:57:35.856200 1731 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:35.856331 1716 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:35.856411 1716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "79585bebb1c94b6ea469ae48b52655bc"
format_stamp: "Formatted at 2026-09-24 02:57:35 on dist-test-slave-sb6p"
server_key: "15c05693dd42da9f27a6855f7797cbb2"
server_key_iv: "376f7a2a9358e7b23d5a676261f1712a"
server_key_version: "encryptionkey@0"
I20260924 02:57:35.856468 1716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:35.867273 1716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:35.869289 1716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:35.869419 1716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:35.869885 1716 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:35.869923 1716 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:35.869953 1716 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:35.869971 1716 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:35.876060 1716 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44619
Sep 24 02:57:35 dist-test-slave-sb6p krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:35.876180 1846 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44619 every 8 connection(s)
I20260924 02:57:35.876416 1716 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 02:57:35.878955 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:35.870328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57487 (local address 127.0.98.62:43965)
0924 02:57:35.870395 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:35.870398 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:35.870534 (+ 136us) server_negotiation.cc:407] Connection header received
0924 02:57:35.871061 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:35.871062 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:35.871080 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:35.871153 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:35.871911 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.872656 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.874097 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.874384 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.874465 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:35.876657 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:35.876658 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:35.876659 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:35.876666 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:35.877742 (+ 1076us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:35.878082 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:35.878083 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:35.878083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:35.878140 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:35.878345 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:35.878346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:35.878347 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:35.878467 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:35.878543 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:35.878796 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 02:57:35.878868 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 02:57:35.879109 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:35.870270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43965 (local address 127.0.98.2:57487)
0924 02:57:35.870410 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:35.870449 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:35.870971 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:35.871231 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:35.871236 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:35.871534 (+ 298us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:35.871816 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:35.871823 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.872743 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.872744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:35.874002 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:35.874005 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.874467 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.874467 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:35.874584 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:35.875057 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:35.875075 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:35.876575 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:35.877812 (+ 1237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:35.877818 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:35.877820 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:35.878018 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:35.878191 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:35.878192 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:35.878192 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:35.878293 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:35.878543 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:35.878549 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:35.878683 (+ 134us) client_negotiation.cc:769] Sending connection context
0924 02:57:35.878769 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 02:57:35.878913 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":74,"threads_started":1}
I20260924 02:57:35.879531 1847 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43965
I20260924 02:57:35.879614 1847 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:35.879782 1847 heartbeater.cc:511] Master 127.0.98.62:43965 requested a full tablet report, sending...
I20260924 02:57:35.880149 1504 ts_manager.cc:194] Registered new tserver with Master: 79585bebb1c94b6ea469ae48b52655bc (127.0.98.2:44619)
I20260924 02:57:35.880586 1504 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:57487
I20260924 02:57:35.882619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 1716
I20260924 02:57:35.882957 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:57:35.883172 392 external_mini_cluster.cc:1428] Setting key 3fea7cb9f768f0b50d8caf755dbde198
I20260924 02:57:35.885712 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:35.880459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39995 (local address 127.0.98.62:43965)
0924 02:57:35.880496 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:35.880498 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:35.880599 (+ 101us) server_negotiation.cc:407] Connection header received
0924 02:57:35.880673 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:35.880674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:35.880682 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:35.880734 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:35.881087 (+ 353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.881525 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.882205 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.882371 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.882430 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:35.883386 (+ 956us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:35.883387 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:35.883387 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:35.883394 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:35.884644 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:35.884994 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:35.884995 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:35.884995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:35.885050 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:35.885214 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:35.885214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:35.885215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:35.885362 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:35.885415 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:35.885558 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 02:57:35.885629 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 02:57:35.885727 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:35.880394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43965 (local address 127.0.98.2:39995)
0924 02:57:35.880539 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:35.880548 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:35.880600 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:35.880750 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:35.880751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:35.880948 (+ 197us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:35.881011 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:35.881013 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.881620 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.881621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:35.882142 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:35.882144 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:35.882479 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:35.882480 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:35.882575 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:35.882978 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:35.882982 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:35.883313 (+ 331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:35.884723 (+ 1410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:35.884724 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:35.884724 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:35.884930 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:35.885104 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:35.885105 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:35.885105 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:35.885160 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:35.885420 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:35.885420 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:35.885470 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 02:57:35.885531 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 02:57:35.885626 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 02:57:35.905565 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43965
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:35.983019 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:35.983223 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:35.983253 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:35.984362 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:35.984400 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:35.984426 1856 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 02:57:35.984448 1856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:57:35.985865 1856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:43965
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1856
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:35.986096 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:35.986291 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:35.991047 1861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:35.991047 1864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:35.991204 1862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:35.991237 1856 server_base.cc:1034] running on GCE node
I20260924 02:57:35.991499 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:35.991693 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:35.992828 1856 hybrid_clock.cc:648] HybridClock initialized: now 1790218655992820 us; error 27 us; skew 500 ppm
Sep 24 02:57:35 dist-test-slave-sb6p krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:35.995157 1856 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 02:57:35.995539 1856 webserver.cc:460] Webserver started at http://127.0.98.3:42249/ using document root <none> and password file <none>
I20260924 02:57:35.995716 1856 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:35.995759 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:35.995841 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:35.996778 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "fdd834f4b5714d10a702fc5559b374d0"
format_stamp: "Formatted at 2026-09-24 02:57:35 on dist-test-slave-sb6p"
server_key: "eaa3fdcb6b320bab9578a2f343495736"
server_key_iv: "76a687a1d72ac7eddb9d4875c2bb7266"
server_key_version: "encryptionkey@0"
I20260924 02:57:35.997057 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fdd834f4b5714d10a702fc5559b374d0"
format_stamp: "Formatted at 2026-09-24 02:57:35 on dist-test-slave-sb6p"
server_key: "eaa3fdcb6b320bab9578a2f343495736"
server_key_iv: "76a687a1d72ac7eddb9d4875c2bb7266"
server_key_version: "encryptionkey@0"
I20260924 02:57:35.998226 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:57:35.998871 1871 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:35.999053 1856 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:35.999128 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "fdd834f4b5714d10a702fc5559b374d0"
format_stamp: "Formatted at 2026-09-24 02:57:35 on dist-test-slave-sb6p"
server_key: "eaa3fdcb6b320bab9578a2f343495736"
server_key_iv: "76a687a1d72ac7eddb9d4875c2bb7266"
server_key_version: "encryptionkey@0"
I20260924 02:57:35.999178 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:36.015863 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:36.017817 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:36.017930 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:36.018581 1856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:36.018635 1856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:36.018664 1856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:36.018685 1856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:36.025300 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34535
Sep 24 02:57:36 dist-test-slave-sb6p krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:36.025455 1986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34535 every 8 connection(s)
I20260924 02:57:36.025691 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 02:57:36.028218 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.019045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42285 (local address 127.0.98.62:43965)
0924 02:57:36.019096 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.019099 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.019320 (+ 221us) server_negotiation.cc:407] Connection header received
0924 02:57:36.020031 (+ 711us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.020032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.020047 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.020078 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:36.021034 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.021770 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.023318 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.023604 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.023669 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.025815 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:36.025817 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:36.025817 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:36.025825 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:36.027120 (+ 1295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:36.027475 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:36.027477 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:36.027477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:36.027556 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:36.027746 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:36.027746 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:36.027747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:36.027847 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:36.027895 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.028094 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.028154 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 02:57:36.028385 1880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.018970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43965 (local address 127.0.98.3:42285)
0924 02:57:36.019185 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:36.019234 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:36.019885 (+ 651us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:36.020277 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:36.020282 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:36.020626 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:36.020923 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.020930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.021870 (+ 940us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.021871 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.023212 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.023215 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.023672 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.023672 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.023780 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.024272 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:36.024280 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:36.025735 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:36.027184 (+ 1449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:36.027190 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:36.027191 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:36.027390 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:36.027612 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:36.027613 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:36.027613 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:36.027700 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:36.027889 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:36.027892 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:36.028000 (+ 108us) client_negotiation.cc:769] Sending connection context
0924 02:57:36.028072 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 02:57:36.028189 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":85,"threads_started":1}
I20260924 02:57:36.028821 1987 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43965
I20260924 02:57:36.028899 1987 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:36.029062 1987 heartbeater.cc:511] Master 127.0.98.62:43965 requested a full tablet report, sending...
I20260924 02:57:36.029400 1503 ts_manager.cc:194] Registered new tserver with Master: fdd834f4b5714d10a702fc5559b374d0 (127.0.98.3:34535)
I20260924 02:57:36.029804 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 1856
I20260924 02:57:36.029848 1503 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:42285
I20260924 02:57:36.029898 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:57:36.030066 392 external_mini_cluster.cc:1428] Setting key c089d7e141182181bf5288d969637d1c
I20260924 02:57:36.030855 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 02:57:36.035184 1993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.029722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43965 (local address 127.0.98.3:54927)
0924 02:57:36.029873 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:36.029884 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:36.029947 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:36.030072 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:36.030072 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:36.030245 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:36.030316 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.030317 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.031160 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.031160 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.031885 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.031887 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.032215 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.032216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.032272 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.032693 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:36.032697 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:36.032934 (+ 237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:36.034347 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:36.034348 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:36.034348 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:36.034536 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:36.034703 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:36.034703 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:36.034704 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:36.034750 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:36.034974 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:36.034974 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:36.035003 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 02:57:36.035046 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 02:57:36.035112 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":65,"threads_started":1}
I20260924 02:57:36.035220 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.029781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54927 (local address 127.0.98.62:43965)
0924 02:57:36.029817 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.029818 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.029947 (+ 129us) server_negotiation.cc:407] Connection header received
0924 02:57:36.029997 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.029998 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.030015 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.030059 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:36.030370 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.031067 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.031964 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.032158 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.032204 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.033005 (+ 801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:36.033006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:36.033007 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:36.033014 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:36.034281 (+ 1267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:36.034592 (+ 311us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:36.034593 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:36.034593 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:36.034649 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:36.034803 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:36.034804 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:36.034804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:36.034921 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:36.034986 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.035072 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.035139 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 02:57:36 dist-test-slave-sb6p krb5kdc[1459](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218656, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 02:57:36 dist-test-slave-sb6p krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218656, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:36.060364 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.053433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59762 (local address 127.0.98.62:43965)
0924 02:57:36.053487 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.053504 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.053515 (+ 11us) server_negotiation.cc:407] Connection header received
0924 02:57:36.053544 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.053546 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.053558 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.053625 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:36.054318 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.055145 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.055877 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.056085 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.056143 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.058247 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:36.058248 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:36.058248 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:36.058255 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:36.059435 (+ 1180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:36.059750 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:36.059750 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:36.059751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:36.059794 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:36.059922 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:36.059922 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:36.059923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:36.060013 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:36.060074 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.060242 (+ 168us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.060299 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 02:57:36.061945 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 02:57:36.063582 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:57:36.069440 2004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.065740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44619 (local address 127.0.0.1:42136)
0924 02:57:36.065908 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:36.065918 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:36.066048 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:36.066218 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:36.066221 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:36.066223 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:36.066284 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.066286 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.067270 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.067270 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.068429 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.068431 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.069163 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.069164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.069235 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.069248 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 02:57:36.069310 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 02:57:36.069362 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":45,"threads_started":1}
I20260924 02:57:36.069454 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.065894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42136 (local address 127.0.98.2:44619)
0924 02:57:36.066066 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.066070 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.066079 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:57:36.066112 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.066113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.066125 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.066198 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:36.066346 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.067200 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.068517 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.069091 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.069157 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.069240 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.069328 (+ 88us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.069368 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":96,"threads_started":1}
I20260924 02:57:36.069829 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.065679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35870 (local address 127.0.98.1:40357)
0924 02:57:36.066164 (+ 485us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.066168 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.066175 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:57:36.066205 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.066207 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.066220 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.066284 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:36.066444 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.067559 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.068762 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.069415 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.069468 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.069532 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.069690 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.069733 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":75,"threads_started":1}
I20260924 02:57:36.069826 2001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.065542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40357 (local address 127.0.0.1:35870)
0924 02:57:36.065712 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:36.065754 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:36.065815 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:36.066298 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:36.066299 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:36.066301 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:36.066366 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.066368 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.067649 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.067649 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.068686 (+ 1037us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.068688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.069558 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.069559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.069614 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.069617 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:36.069671 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 02:57:36.069739 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"mutex_wait_us":17,"thread_start_us":63,"threads_started":1}
I20260924 02:57:36.069826 2002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.065549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34535 (local address 127.0.0.1:38486)
0924 02:57:36.065897 (+ 348us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:36.065903 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:36.065928 (+ 25us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:36.066573 (+ 645us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:36.066574 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:36.066576 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:36.066608 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.066609 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.067768 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.067769 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.068571 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.068573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.069543 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.069544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.069615 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.069620 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 02:57:36.069671 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 02:57:36.069739 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":18,"thread_start_us":60,"threads_started":1}
I20260924 02:57:36.070089 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.065708 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38486 (local address 127.0.98.3:34535)
0924 02:57:36.066398 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.066402 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.066410 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:57:36.066443 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.066445 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.066459 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.066674 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:36.066724 (+ 50us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.067703 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.068649 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.069353 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.069420 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.069506 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.069953 (+ 447us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.069998 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":658,"thread_start_us":77,"threads_started":1}
I20260924 02:57:36.070403 1781 tablet_service.cc:1467] Processing CreateTablet for tablet f9846696f9d345449acab85a23a65431 (DEFAULT_TABLE table=test-table [id=380e4d86cbd7466dbe348a42ebe987ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:36.070806 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9846696f9d345449acab85a23a65431. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:36.070700 1641 tablet_service.cc:1467] Processing CreateTablet for tablet f9846696f9d345449acab85a23a65431 (DEFAULT_TABLE table=test-table [id=380e4d86cbd7466dbe348a42ebe987ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:36.070724 1910 tablet_service.cc:1467] Processing CreateTablet for tablet f9846696f9d345449acab85a23a65431 (DEFAULT_TABLE table=test-table [id=380e4d86cbd7466dbe348a42ebe987ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:36.070997 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9846696f9d345449acab85a23a65431. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:36.071012 1910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9846696f9d345449acab85a23a65431. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:36.073535 2007 tablet_bootstrap.cc:492] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0: Bootstrap starting.
I20260924 02:57:36.074080 2007 tablet_bootstrap.cc:654] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:36.074345 2007 log.cc:824] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:36.074606 2008 tablet_bootstrap.cc:492] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc: Bootstrap starting.
I20260924 02:57:36.075219 2008 tablet_bootstrap.cc:654] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:36.075294 2007 tablet_bootstrap.cc:492] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0: No bootstrap required, opened a new log
I20260924 02:57:36.075359 2007 ts_tablet_manager.cc:1397] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:36.075520 2008 log.cc:824] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:36.075866 2009 tablet_bootstrap.cc:492] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6: Bootstrap starting.
I20260924 02:57:36.076193 2008 tablet_bootstrap.cc:492] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc: No bootstrap required, opened a new log
I20260924 02:57:36.076251 2008 ts_tablet_manager.cc:1397] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:36.076447 2009 tablet_bootstrap.cc:654] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:36.076753 2009 log.cc:824] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:36.077221 2007 raft_consensus.cc:348] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } }
I20260924 02:57:36.077382 2007 raft_consensus.cc:374] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:36.077421 2007 raft_consensus.cc:729] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd834f4b5714d10a702fc5559b374d0, State: Initialized, Role: FOLLOWER
I20260924 02:57:36.077472 2009 tablet_bootstrap.cc:492] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6: No bootstrap required, opened a new log
I20260924 02:57:36.077526 2009 ts_tablet_manager.cc:1397] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:36.077538 2007 consensus_queue.cc:260] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } }
I20260924 02:57:36.077790 2007 ts_tablet_manager.cc:1428] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 02:57:36.077903 1987 heartbeater.cc:503] Master 127.0.98.62:43965 was elected leader, sending a full tablet report...
I20260924 02:57:36.077968 2008 raft_consensus.cc:348] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } }
I20260924 02:57:36.078122 2008 raft_consensus.cc:374] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:36.078233 2008 raft_consensus.cc:729] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79585bebb1c94b6ea469ae48b52655bc, State: Initialized, Role: FOLLOWER
I20260924 02:57:36.078383 2008 consensus_queue.cc:260] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } }
I20260924 02:57:36.078732 1847 heartbeater.cc:503] Master 127.0.98.62:43965 was elected leader, sending a full tablet report...
I20260924 02:57:36.078971 2008 ts_tablet_manager.cc:1428] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260924 02:57:36.079368 2009 raft_consensus.cc:348] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } }
I20260924 02:57:36.079517 2009 raft_consensus.cc:374] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:36.079552 2009 raft_consensus.cc:729] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f632640b969c4ac4954fb8c033e101f6, State: Initialized, Role: FOLLOWER
I20260924 02:57:36.079676 2009 consensus_queue.cc:260] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } }
I20260924 02:57:36.079861 2009 ts_tablet_manager.cc:1428] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 02:57:36.079911 1707 heartbeater.cc:503] Master 127.0.98.62:43965 was elected leader, sending a full tablet report...
W20260924 02:57:36.127074 1848 tablet.cc:2369] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 02:57:36.146693 2014 raft_consensus.cc:482] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:57:36.146807 2014 raft_consensus.cc:504] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } }
I20260924 02:57:36.147105 2014 leader_election.cc:288] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f632640b969c4ac4954fb8c033e101f6 (127.0.98.1:40357), fdd834f4b5714d10a702fc5559b374d0 (127.0.98.3:34535)
I20260924 02:57:36.150835 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.147406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36193 (local address 127.0.98.3:34535)
0924 02:57:36.147446 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.147448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.147457 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:57:36.147483 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.147483 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.147493 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.147554 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:36.147694 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.148477 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.149902 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.150451 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.150547 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.150559 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.150730 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.150765 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 02:57:36.150835 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.147379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59531 (local address 127.0.98.1:40357)
0924 02:57:36.147421 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.147423 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.147558 (+ 135us) server_negotiation.cc:407] Connection header received
0924 02:57:36.147590 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.147591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.147602 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.147646 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:36.147778 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.148454 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.149902 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.150451 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.150557 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.150566 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.150730 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.150764 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 02:57:36.150867 1740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.147286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34535 (local address 127.0.98.2:36193)
0924 02:57:36.147340 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:36.147352 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:36.147397 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:36.147565 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:36.147566 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:36.147567 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:36.147631 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.147632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.148557 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.148558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.149811 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.149814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.150560 (+ 746us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.150561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.150655 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.150660 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 02:57:36.150719 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 02:57:36.150760 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":47}
I20260924 02:57:36.150861 2016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.147297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40357 (local address 127.0.98.2:59531)
0924 02:57:36.147484 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:36.147492 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:36.147548 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:36.147660 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:36.147661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:36.147663 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:36.147723 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.147725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.148568 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.148568 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.149811 (+ 1243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.149814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.150573 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.150574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.150654 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.150661 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 02:57:36.150719 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 02:57:36.150760 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":25,"thread_start_us":103,"threads_started":1}
I20260924 02:57:36.151219 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9846696f9d345449acab85a23a65431" candidate_uuid: "79585bebb1c94b6ea469ae48b52655bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd834f4b5714d10a702fc5559b374d0" is_pre_election: true
I20260924 02:57:36.151381 1941 raft_consensus.cc:2454] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79585bebb1c94b6ea469ae48b52655bc in term 0.
I20260924 02:57:36.151348 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9846696f9d345449acab85a23a65431" candidate_uuid: "79585bebb1c94b6ea469ae48b52655bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f632640b969c4ac4954fb8c033e101f6" is_pre_election: true
I20260924 02:57:36.151496 1661 raft_consensus.cc:2454] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79585bebb1c94b6ea469ae48b52655bc in term 0.
I20260924 02:57:36.151608 1735 leader_election.cc:302] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79585bebb1c94b6ea469ae48b52655bc, fdd834f4b5714d10a702fc5559b374d0; no voters:
I20260924 02:57:36.151757 2014 raft_consensus.cc:2781] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:57:36.151816 2014 raft_consensus.cc:482] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:57:36.151832 2014 raft_consensus.cc:3037] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:36.152751 2014 raft_consensus.cc:504] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } }
I20260924 02:57:36.152925 2014 leader_election.cc:288] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [CANDIDATE]: Term 1 election: Requested vote from peers f632640b969c4ac4954fb8c033e101f6 (127.0.98.1:40357), fdd834f4b5714d10a702fc5559b374d0 (127.0.98.3:34535)
I20260924 02:57:36.153009 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9846696f9d345449acab85a23a65431" candidate_uuid: "79585bebb1c94b6ea469ae48b52655bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f632640b969c4ac4954fb8c033e101f6"
I20260924 02:57:36.153013 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9846696f9d345449acab85a23a65431" candidate_uuid: "79585bebb1c94b6ea469ae48b52655bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd834f4b5714d10a702fc5559b374d0"
I20260924 02:57:36.153079 1941 raft_consensus.cc:3037] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:36.153079 1661 raft_consensus.cc:3037] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:36.153879 1661 raft_consensus.cc:2454] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79585bebb1c94b6ea469ae48b52655bc in term 1.
I20260924 02:57:36.153879 1941 raft_consensus.cc:2454] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79585bebb1c94b6ea469ae48b52655bc in term 1.
I20260924 02:57:36.154042 1735 leader_election.cc:302] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79585bebb1c94b6ea469ae48b52655bc, fdd834f4b5714d10a702fc5559b374d0; no voters:
I20260924 02:57:36.154138 2014 raft_consensus.cc:2781] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:36.154313 2014 raft_consensus.cc:686] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [term 1 LEADER]: Becoming Leader. State: Replica: 79585bebb1c94b6ea469ae48b52655bc, State: Running, Role: LEADER
I20260924 02:57:36.154397 2014 consensus_queue.cc:237] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } }
I20260924 02:57:36.155254 1503 catalog_manager.cc:5495] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 79585bebb1c94b6ea469ae48b52655bc (127.0.98.2). New cstate: current_term: 1 leader_uuid: "79585bebb1c94b6ea469ae48b52655bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } health_report { overall_health: HEALTHY } } }
I20260924 02:57:36.165851 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.163380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42138 (local address 127.0.98.2:44619)
0924 02:57:36.163432 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.163434 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.163445 (+ 11us) server_negotiation.cc:407] Connection header received
0924 02:57:36.163552 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.163553 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.163569 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.163622 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:36.163851 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.164299 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.165173 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.165348 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.165392 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.165553 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:36.165662 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:36.165717 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.165773 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.165789 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 02:57:36.167520 1941 raft_consensus.cc:1261] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0 [term 1 FOLLOWER]: Refusing update from remote peer 79585bebb1c94b6ea469ae48b52655bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:36.167531 1661 raft_consensus.cc:1261] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6 [term 1 FOLLOWER]: Refusing update from remote peer 79585bebb1c94b6ea469ae48b52655bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:36.167867 2017 consensus_queue.cc:1035] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:36.168004 2014 consensus_queue.cc:1035] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:36.170250 2020 mvcc.cc:205] Tried to move back new op lower bound from 7332735615659843584 to 7332735615609131008. Current Snapshot: MvccSnapshot[applied={T|T < 7332735615659843584}]
I20260924 02:57:36.170588 2022 mvcc.cc:205] Tried to move back new op lower bound from 7332735615659843584 to 7332735615609131008. Current Snapshot: MvccSnapshot[applied={T|T < 7332735615659843584}]
I20260924 02:57:36.170689 2021 mvcc.cc:205] Tried to move back new op lower bound from 7332735615659843584 to 7332735615609131008. Current Snapshot: MvccSnapshot[applied={T|T < 7332735615659843584}]
I20260924 02:57:36.173823 1503 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:36.173900 1503 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:36.174605 1503 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8: Sending DeleteTablet for 3 replicas of tablet f9846696f9d345449acab85a23a65431
I20260924 02:57:36.174962 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet f9846696f9d345449acab85a23a65431 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35870
I20260924 02:57:36.175034 1910 tablet_service.cc:1514] Processing DeleteTablet for tablet f9846696f9d345449acab85a23a65431 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38486
I20260924 02:57:36.175122 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 02:57:36.175174 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet f9846696f9d345449acab85a23a65431 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42136
I20260924 02:57:36.175232 2030 tablet_replica.cc:330] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6: stopping tablet replica
W20260924 02:57:36.175256 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:57:36.175318 2030 raft_consensus.cc:2229] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:36.175803 2030 raft_consensus.cc:2258] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:36.176102 2032 tablet_replica.cc:330] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc: stopping tablet replica
I20260924 02:57:36.176417 2030 ts_tablet_manager.cc:1905] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:36.176568 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 8c235ac8df554784a382f023fd26f7d4 (DEFAULT_TABLE table=test-table [id=c1af280bd1864c32a57bd9b531fbf205]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:36.176636 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 8c235ac8df554784a382f023fd26f7d4 (DEFAULT_TABLE table=test-table [id=c1af280bd1864c32a57bd9b531fbf205]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:36.176677 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c235ac8df554784a382f023fd26f7d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:36.176729 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c235ac8df554784a382f023fd26f7d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:36.177937 2009 tablet_bootstrap.cc:492] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6: Bootstrap starting.
I20260924 02:57:36.177937 2008 tablet_bootstrap.cc:492] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc: Bootstrap starting.
I20260924 02:57:36.178318 2009 tablet_bootstrap.cc:654] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:36.178324 2008 tablet_bootstrap.cc:654] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:36.178879 2009 tablet_bootstrap.cc:492] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6: No bootstrap required, opened a new log
I20260924 02:57:36.178942 2009 ts_tablet_manager.cc:1397] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:36.178937 1910 tablet_service.cc:1467] Processing CreateTablet for tablet 8c235ac8df554784a382f023fd26f7d4 (DEFAULT_TABLE table=test-table [id=c1af280bd1864c32a57bd9b531fbf205]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:36.178894 2032 raft_consensus.cc:2229] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [term 1 LEADER]: Raft consensus shutting down.
I20260924 02:57:36.179046 1910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c235ac8df554784a382f023fd26f7d4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:36.179101 2009 raft_consensus.cc:348] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.179194 2009 raft_consensus.cc:374] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:36.179220 2009 raft_consensus.cc:729] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f632640b969c4ac4954fb8c033e101f6, State: Initialized, Role: FOLLOWER
I20260924 02:57:36.179257 2009 consensus_queue.cc:260] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.179339 2009 ts_tablet_manager.cc:1428] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:36.179281 2032 raft_consensus.cc:2258] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:36.179606 2031 tablet_replica.cc:330] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0: stopping tablet replica
I20260924 02:57:36.179745 2031 raft_consensus.cc:2229] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:36.179901 2031 raft_consensus.cc:2258] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:36.180352 2030 ts_tablet_manager.cc:1918] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:57:36.180420 2030 log.cc:1196] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/wal/wals/f9846696f9d345449acab85a23a65431
I20260924 02:57:36.180492 2007 tablet_bootstrap.cc:492] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0: Bootstrap starting.
I20260924 02:57:36.180673 2030 ts_tablet_manager.cc:1939] T f9846696f9d345449acab85a23a65431 P f632640b969c4ac4954fb8c033e101f6: Deleting consensus metadata
I20260924 02:57:36.180841 2007 tablet_bootstrap.cc:654] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:36.181071 2008 tablet_bootstrap.cc:492] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc: No bootstrap required, opened a new log
I20260924 02:57:36.181123 2008 ts_tablet_manager.cc:1397] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 02:57:36.181278 2008 raft_consensus.cc:348] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.181368 2008 raft_consensus.cc:374] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:36.181392 2008 raft_consensus.cc:729] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79585bebb1c94b6ea469ae48b52655bc, State: Initialized, Role: FOLLOWER
I20260924 02:57:36.181452 2008 consensus_queue.cc:260] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.181526 2008 ts_tablet_manager.cc:1428] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:36.181684 1490 catalog_manager.cc:4841] TS f632640b969c4ac4954fb8c033e101f6 (127.0.98.1:40357): tablet f9846696f9d345449acab85a23a65431 (table test-table [id=380e4d86cbd7466dbe348a42ebe987ba]) successfully deleted
I20260924 02:57:36.182113 2032 ts_tablet_manager.cc:1905] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:36.180382 2031 ts_tablet_manager.cc:1905] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:36.183575 2032 ts_tablet_manager.cc:1918] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:57:36.183632 2032 log.cc:1196] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/f9846696f9d345449acab85a23a65431
I20260924 02:57:36.183847 2032 ts_tablet_manager.cc:1939] T f9846696f9d345449acab85a23a65431 P 79585bebb1c94b6ea469ae48b52655bc: Deleting consensus metadata
I20260924 02:57:36.183933 2031 ts_tablet_manager.cc:1918] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:57:36.184000 2031 log.cc:1196] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/f9846696f9d345449acab85a23a65431
I20260924 02:57:36.184218 2031 ts_tablet_manager.cc:1939] T f9846696f9d345449acab85a23a65431 P fdd834f4b5714d10a702fc5559b374d0: Deleting consensus metadata
I20260924 02:57:36.184204 1491 catalog_manager.cc:4841] TS 79585bebb1c94b6ea469ae48b52655bc (127.0.98.2:44619): tablet f9846696f9d345449acab85a23a65431 (table test-table [id=380e4d86cbd7466dbe348a42ebe987ba]) successfully deleted
I20260924 02:57:36.184569 1491 catalog_manager.cc:4841] TS fdd834f4b5714d10a702fc5559b374d0 (127.0.98.3:34535): tablet f9846696f9d345449acab85a23a65431 (table test-table [id=380e4d86cbd7466dbe348a42ebe987ba]) successfully deleted
I20260924 02:57:36.184778 2007 tablet_bootstrap.cc:492] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0: No bootstrap required, opened a new log
I20260924 02:57:36.184816 2007 ts_tablet_manager.cc:1397] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260924 02:57:36.184949 2007 raft_consensus.cc:348] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.185011 2007 raft_consensus.cc:374] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:36.185030 2007 raft_consensus.cc:729] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd834f4b5714d10a702fc5559b374d0, State: Initialized, Role: FOLLOWER
I20260924 02:57:36.185068 2007 consensus_queue.cc:260] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.185141 2007 ts_tablet_manager.cc:1428] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:36.190927 2015 raft_consensus.cc:482] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:57:36.190997 2015 raft_consensus.cc:504] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.191298 2015 leader_election.cc:288] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 79585bebb1c94b6ea469ae48b52655bc (127.0.98.2:44619), fdd834f4b5714d10a702fc5559b374d0 (127.0.98.3:34535)
I20260924 02:57:36.194250 2005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.191729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36999 (local address 127.0.98.3:34535)
0924 02:57:36.191762 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.191764 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.191808 (+ 44us) server_negotiation.cc:407] Connection header received
0924 02:57:36.191861 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.191862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.191869 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.191912 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:36.192046 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.192509 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.193577 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.193930 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.194001 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.194011 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.194152 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.194187 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 02:57:36.194283 2036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.191593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34535 (local address 127.0.98.1:36999)
0924 02:57:36.191762 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:36.191772 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:36.191819 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:36.191913 (+ 94us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:36.191913 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:36.191915 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:36.191991 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.191993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.192631 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.192632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.193494 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.193496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.194017 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.194018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.194085 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.194090 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 02:57:36.194139 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 02:57:36.194176 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"mutex_wait_us":38,"thread_start_us":59,"threads_started":1}
I20260924 02:57:36.194779 1600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.191473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44619 (local address 127.0.98.1:38905)
0924 02:57:36.191523 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:36.191536 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:36.191584 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:36.191712 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:36.191713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:36.191714 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:36.191776 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.191778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.192674 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.192675 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.193860 (+ 1185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.193863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.194503 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.194504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.194620 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.194624 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:57:36.194681 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 02:57:36.194717 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":33}
I20260924 02:57:36.194746 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c235ac8df554784a382f023fd26f7d4" candidate_uuid: "f632640b969c4ac4954fb8c033e101f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd834f4b5714d10a702fc5559b374d0" is_pre_election: true
I20260924 02:57:36.194787 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.191520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38905 (local address 127.0.98.2:44619)
0924 02:57:36.191569 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.191570 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.191580 (+ 10us) server_negotiation.cc:407] Connection header received
0924 02:57:36.191639 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.191640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.191648 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.191705 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:36.191852 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.192587 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.193947 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.194427 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.194491 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.194499 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.194696 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.194731 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 02:57:36.194824 1941 raft_consensus.cc:2454] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f632640b969c4ac4954fb8c033e101f6 in term 0.
I20260924 02:57:36.194999 1595 leader_election.cc:302] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f632640b969c4ac4954fb8c033e101f6, fdd834f4b5714d10a702fc5559b374d0; no voters:
I20260924 02:57:36.195125 2015 raft_consensus.cc:2781] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:57:36.195171 2015 raft_consensus.cc:482] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:57:36.195196 2015 raft_consensus.cc:3037] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:36.195206 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c235ac8df554784a382f023fd26f7d4" candidate_uuid: "f632640b969c4ac4954fb8c033e101f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79585bebb1c94b6ea469ae48b52655bc" is_pre_election: true
I20260924 02:57:36.195317 1801 raft_consensus.cc:2454] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f632640b969c4ac4954fb8c033e101f6 in term 0.
I20260924 02:57:36.196008 2015 raft_consensus.cc:504] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.196139 2015 leader_election.cc:288] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [CANDIDATE]: Term 1 election: Requested vote from peers 79585bebb1c94b6ea469ae48b52655bc (127.0.98.2:44619), fdd834f4b5714d10a702fc5559b374d0 (127.0.98.3:34535)
I20260924 02:57:36.196300 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c235ac8df554784a382f023fd26f7d4" candidate_uuid: "f632640b969c4ac4954fb8c033e101f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd834f4b5714d10a702fc5559b374d0"
I20260924 02:57:36.196359 1941 raft_consensus.cc:3037] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:36.196933 1941 raft_consensus.cc:2454] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f632640b969c4ac4954fb8c033e101f6 in term 1.
I20260924 02:57:36.197042 1801 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c235ac8df554784a382f023fd26f7d4" candidate_uuid: "f632640b969c4ac4954fb8c033e101f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "79585bebb1c94b6ea469ae48b52655bc"
I20260924 02:57:36.197079 1595 leader_election.cc:302] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: f632640b969c4ac4954fb8c033e101f6, fdd834f4b5714d10a702fc5559b374d0; no voters:
I20260924 02:57:36.197108 1801 raft_consensus.cc:3037] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:36.197152 2015 raft_consensus.cc:2781] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:36.197280 2015 raft_consensus.cc:686] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [term 1 LEADER]: Becoming Leader. State: Replica: f632640b969c4ac4954fb8c033e101f6, State: Running, Role: LEADER
I20260924 02:57:36.197343 2015 consensus_queue.cc:237] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.197834 1801 raft_consensus.cc:2454] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f632640b969c4ac4954fb8c033e101f6 in term 1.
I20260924 02:57:36.198271 1503 catalog_manager.cc:5495] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to f632640b969c4ac4954fb8c033e101f6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "f632640b969c4ac4954fb8c033e101f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } health_report { overall_health: UNKNOWN } } }
W20260924 02:57:36.233777 1708 tablet.cc:2369] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 02:57:36 dist-test-slave-sb6p krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218655, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:36.238171 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.231196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59768 (local address 127.0.98.62:43965)
0924 02:57:36.231255 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.231257 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.231340 (+ 83us) server_negotiation.cc:407] Connection header received
0924 02:57:36.231387 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.231388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.231399 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.231469 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:36.231873 (+ 404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.232596 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.233261 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.233545 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.233620 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.235696 (+ 2076us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:36.235697 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:36.235697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:36.235703 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:36.236944 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:36.237395 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:36.237396 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:36.237397 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:36.237453 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:36.237649 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:36.237650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:36.237650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:36.237750 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:36.237796 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.237946 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.238098 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":33}
I20260924 02:57:36.238171 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.231125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43965 (local address 127.0.0.1:59768)
0924 02:57:36.231270 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:36.231280 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:36.231343 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:36.231474 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:36.231474 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:36.231756 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:36.231800 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.231802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.232667 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.232667 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.233188 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.233189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.233613 (+ 424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.233614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.233651 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.234174 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:36.234179 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:36.235618 (+ 1439us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:36.237015 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:36.237022 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:36.237024 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:36.237306 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:36.237519 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:36.237520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:36.237520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:36.237590 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:36.237801 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:36.237807 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:36.237857 (+ 50us) client_negotiation.cc:769] Sending connection context
0924 02:57:36.237928 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 02:57:36.238048 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
I20260924 02:57:36.240195 1503 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59768:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 02:57:36.240401 1503 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:57:36.241806 1641 tablet_service.cc:1467] Processing CreateTablet for tablet 894887d6feeb48719e3db7df44152dd6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b3b46d0cdc2e4e88a5af3af4f406979a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 02:57:36.241881 1781 tablet_service.cc:1467] Processing CreateTablet for tablet 894887d6feeb48719e3db7df44152dd6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b3b46d0cdc2e4e88a5af3af4f406979a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 02:57:36.241961 1641 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 894887d6feeb48719e3db7df44152dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:36.241935 1910 tablet_service.cc:1467] Processing CreateTablet for tablet 894887d6feeb48719e3db7df44152dd6 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=b3b46d0cdc2e4e88a5af3af4f406979a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 02:57:36.242013 1781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 894887d6feeb48719e3db7df44152dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:36.242074 1910 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 894887d6feeb48719e3db7df44152dd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:36.243409 2009 tablet_bootstrap.cc:492] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6: Bootstrap starting.
I20260924 02:57:36.243492 2008 tablet_bootstrap.cc:492] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc: Bootstrap starting.
I20260924 02:57:36.243799 2009 tablet_bootstrap.cc:654] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:36.243834 2008 tablet_bootstrap.cc:654] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:36.244163 2007 tablet_bootstrap.cc:492] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0: Bootstrap starting.
I20260924 02:57:36.244606 2007 tablet_bootstrap.cc:654] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:36.245270 2007 tablet_bootstrap.cc:492] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0: No bootstrap required, opened a new log
I20260924 02:57:36.245303 2009 tablet_bootstrap.cc:492] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6: No bootstrap required, opened a new log
I20260924 02:57:36.245313 2007 ts_tablet_manager.cc:1397] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:36.245347 2009 ts_tablet_manager.cc:1397] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:36.245455 2007 raft_consensus.cc:348] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.245525 2007 raft_consensus.cc:374] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:36.245544 2007 raft_consensus.cc:729] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fdd834f4b5714d10a702fc5559b374d0, State: Initialized, Role: FOLLOWER
I20260924 02:57:36.245502 2009 raft_consensus.cc:348] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.245573 2009 raft_consensus.cc:374] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:36.245589 2007 consensus_queue.cc:260] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.245621 2009 raft_consensus.cc:729] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f632640b969c4ac4954fb8c033e101f6, State: Initialized, Role: FOLLOWER
I20260924 02:57:36.245669 2007 ts_tablet_manager.cc:1428] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:36.245668 2009 consensus_queue.cc:260] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.245759 2009 ts_tablet_manager.cc:1428] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:36.245815 2013 tablet_replica.cc:439] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } }
I20260924 02:57:36.245893 2013 tablet_replica.cc:442] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:36.245908 2015 tablet_replica.cc:439] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } }
I20260924 02:57:36.245996 2015 tablet_replica.cc:442] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:36.246918 2008 tablet_bootstrap.cc:492] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc: No bootstrap required, opened a new log
I20260924 02:57:36.246971 2008 ts_tablet_manager.cc:1397] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260924 02:57:36.247097 2008 raft_consensus.cc:348] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.247157 2008 raft_consensus.cc:374] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:36.247175 2008 raft_consensus.cc:729] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79585bebb1c94b6ea469ae48b52655bc, State: Initialized, Role: FOLLOWER
I20260924 02:57:36.247215 2008 consensus_queue.cc:260] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.247298 2008 ts_tablet_manager.cc:1428] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:36.247392 2025 tablet_replica.cc:439] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } }
I20260924 02:57:36.247442 2025 tablet_replica.cc:442] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:36.259423 2025 raft_consensus.cc:482] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:57:36.259500 2025 raft_consensus.cc:504] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.259691 2025 leader_election.cc:288] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f632640b969c4ac4954fb8c033e101f6 (127.0.98.1:40357), fdd834f4b5714d10a702fc5559b374d0 (127.0.98.3:34535)
I20260924 02:57:36.259864 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "894887d6feeb48719e3db7df44152dd6" candidate_uuid: "79585bebb1c94b6ea469ae48b52655bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd834f4b5714d10a702fc5559b374d0" is_pre_election: true
I20260924 02:57:36.259868 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "894887d6feeb48719e3db7df44152dd6" candidate_uuid: "79585bebb1c94b6ea469ae48b52655bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f632640b969c4ac4954fb8c033e101f6" is_pre_election: true
I20260924 02:57:36.259954 1941 raft_consensus.cc:2454] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79585bebb1c94b6ea469ae48b52655bc in term 0.
I20260924 02:57:36.259968 1661 raft_consensus.cc:2454] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 79585bebb1c94b6ea469ae48b52655bc in term 0.
I20260924 02:57:36.260110 1734 leader_election.cc:302] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 79585bebb1c94b6ea469ae48b52655bc, f632640b969c4ac4954fb8c033e101f6, fdd834f4b5714d10a702fc5559b374d0; no voters:
I20260924 02:57:36.260201 2025 raft_consensus.cc:2781] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:57:36.260237 2025 raft_consensus.cc:482] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:57:36.260254 2025 raft_consensus.cc:3037] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:36.261392 2025 raft_consensus.cc:504] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.261528 2025 leader_election.cc:288] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [CANDIDATE]: Term 1 election: Requested vote from peers f632640b969c4ac4954fb8c033e101f6 (127.0.98.1:40357), fdd834f4b5714d10a702fc5559b374d0 (127.0.98.3:34535)
I20260924 02:57:36.261685 1941 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "894887d6feeb48719e3db7df44152dd6" candidate_uuid: "79585bebb1c94b6ea469ae48b52655bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fdd834f4b5714d10a702fc5559b374d0"
I20260924 02:57:36.261755 1941 raft_consensus.cc:3037] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:36.261780 1661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "894887d6feeb48719e3db7df44152dd6" candidate_uuid: "79585bebb1c94b6ea469ae48b52655bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f632640b969c4ac4954fb8c033e101f6"
I20260924 02:57:36.261840 1661 raft_consensus.cc:3037] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:36.262444 1941 raft_consensus.cc:2454] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79585bebb1c94b6ea469ae48b52655bc in term 1.
I20260924 02:57:36.262487 1661 raft_consensus.cc:2454] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 79585bebb1c94b6ea469ae48b52655bc in term 1.
I20260924 02:57:36.262620 1735 leader_election.cc:302] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 79585bebb1c94b6ea469ae48b52655bc, fdd834f4b5714d10a702fc5559b374d0; no voters:
I20260924 02:57:36.262714 2025 raft_consensus.cc:2781] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:36.262756 2025 raft_consensus.cc:686] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [term 1 LEADER]: Becoming Leader. State: Replica: 79585bebb1c94b6ea469ae48b52655bc, State: Running, Role: LEADER
I20260924 02:57:36.262804 2025 consensus_queue.cc:237] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } }
I20260924 02:57:36.263065 2014 tablet_replica.cc:439] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc: TxnStatusTablet state changed. Reason: New leader 79585bebb1c94b6ea469ae48b52655bc. Latest consensus state: current_term: 1 leader_uuid: "79585bebb1c94b6ea469ae48b52655bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } }
I20260924 02:57:36.263152 2014 tablet_replica.cc:442] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc: This TxnStatusTablet replica's current role is: LEADER
I20260924 02:57:36.263363 1502 catalog_manager.cc:5495] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 79585bebb1c94b6ea469ae48b52655bc (127.0.98.2). New cstate: current_term: 1 leader_uuid: "79585bebb1c94b6ea469ae48b52655bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } health_report { overall_health: UNKNOWN } } }
I20260924 02:57:36.266634 2049 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 02:57:36.266724 2049 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 02:57:36.271150 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.268727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42146 (local address 127.0.98.2:44619)
0924 02:57:36.268767 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.268769 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.268845 (+ 76us) server_negotiation.cc:407] Connection header received
0924 02:57:36.268929 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.268930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.268939 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.268980 (+ 41us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:36.269101 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.269542 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.270559 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.270748 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.270796 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.270972 (+ 176us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:36.270997 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:36.271026 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.271085 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.271099 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 02:57:36.271399 2044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.268673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44619 (local address 127.0.0.1:42146)
0924 02:57:36.268797 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:36.268810 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:36.268865 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:36.268995 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:36.268996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:36.268997 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 02:57:36.269048 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.269050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.269620 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.269621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.270430 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.270434 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.270812 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.270813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.270882 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.270900 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 02:57:36.271033 (+ 133us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 02:57:36.271038 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 02:57:36.271082 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 02:57:36.271136 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83}
I20260924 02:57:36.272282 1941 raft_consensus.cc:1261] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0 [term 1 FOLLOWER]: Refusing update from remote peer 79585bebb1c94b6ea469ae48b52655bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:36.272315 1661 raft_consensus.cc:1261] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6 [term 1 FOLLOWER]: Refusing update from remote peer 79585bebb1c94b6ea469ae48b52655bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:36.272451 2025 consensus_queue.cc:1035] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:36.272590 2025 consensus_queue.cc:1035] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:36.273119 2015 tablet_replica.cc:439] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6: TxnStatusTablet state changed. Reason: New leader 79585bebb1c94b6ea469ae48b52655bc. Latest consensus state: current_term: 1 leader_uuid: "79585bebb1c94b6ea469ae48b52655bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } }
I20260924 02:57:36.273213 2015 tablet_replica.cc:442] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:36.273186 2013 tablet_replica.cc:439] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0: TxnStatusTablet state changed. Reason: New leader 79585bebb1c94b6ea469ae48b52655bc. Latest consensus state: current_term: 1 leader_uuid: "79585bebb1c94b6ea469ae48b52655bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } }
I20260924 02:57:36.273259 2013 tablet_replica.cc:442] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:36.273974 2025 tablet_replica.cc:439] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79585bebb1c94b6ea469ae48b52655bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } }
I20260924 02:57:36.274046 2025 tablet_replica.cc:442] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc: This TxnStatusTablet replica's current role is: LEADER
I20260924 02:57:36.274091 2025 tablet_replica.cc:439] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "79585bebb1c94b6ea469ae48b52655bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } }
I20260924 02:57:36.274124 2025 tablet_replica.cc:442] T 894887d6feeb48719e3db7df44152dd6 P 79585bebb1c94b6ea469ae48b52655bc: This TxnStatusTablet replica's current role is: LEADER
I20260924 02:57:36.274678 2013 tablet_replica.cc:439] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79585bebb1c94b6ea469ae48b52655bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } }
I20260924 02:57:36.274760 2013 tablet_replica.cc:442] T 894887d6feeb48719e3db7df44152dd6 P fdd834f4b5714d10a702fc5559b374d0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:36.275298 2015 tablet_replica.cc:439] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "79585bebb1c94b6ea469ae48b52655bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f632640b969c4ac4954fb8c033e101f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40357 } } peers { permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 } } peers { permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 } } }
I20260924 02:57:36.275384 2015 tablet_replica.cc:442] T 894887d6feeb48719e3db7df44152dd6 P f632640b969c4ac4954fb8c033e101f6: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 02:57:36.276216 1988 tablet.cc:2369] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 02:57:36.278038 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.275847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.0.98.1:40357)
0924 02:57:36.275884 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.275886 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.275893 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:57:36.275958 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.275959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.275978 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.276036 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:36.276173 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.276681 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.277350 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.277549 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.277601 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.277745 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:36.277846 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:36.277903 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.277944 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.277965 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 02:57:36.282452 2006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.280433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59653 (local address 127.0.98.2:44619)
0924 02:57:36.280471 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.280473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.280480 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:57:36.280529 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.280529 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.280540 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.280589 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:36.280704 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.281138 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.281844 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.282036 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.282087 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.282237 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:36.282268 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:36.282311 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.282367 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.282385 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 02:57:36.282678 2055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.280286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44619 (local address 127.0.98.1:59653)
0924 02:57:36.280424 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:36.280434 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:36.280484 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:36.280591 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:36.280592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:36.280593 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 02:57:36.280650 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.280652 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.281202 (+ 550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.281202 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.281775 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.281776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.282090 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.282091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.282160 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.282174 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 02:57:36.282316 (+ 142us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 02:57:36.282317 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 02:57:36.282363 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 02:57:36.282402 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":81,"threads_started":1}
I20260924 02:57:36.284583 1941 raft_consensus.cc:1261] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0 [term 1 FOLLOWER]: Refusing update from remote peer f632640b969c4ac4954fb8c033e101f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:36.284689 1801 raft_consensus.cc:1261] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc [term 1 FOLLOWER]: Refusing update from remote peer f632640b969c4ac4954fb8c033e101f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:36.284792 2015 consensus_queue.cc:1035] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fdd834f4b5714d10a702fc5559b374d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:36.284894 2037 consensus_queue.cc:1035] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "79585bebb1c94b6ea469ae48b52655bc" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44619 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:36.292867 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.290613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54745 (local address 127.0.98.1:40357)
0924 02:57:36.290648 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.290650 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.290658 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:57:36.290711 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.290711 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.290719 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.290759 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:36.290855 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.291300 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.292320 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.292517 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.292564 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.292711 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:36.292734 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:36.292764 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.292803 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.292817 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 02:57:36.292914 1853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.290543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40357 (local address 127.0.98.2:54745)
0924 02:57:36.290597 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:36.290610 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:36.290663 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:36.290763 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:36.290763 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:36.290765 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 02:57:36.290803 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.290805 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.291367 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.291368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.292250 (+ 882us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:36.292253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.292574 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.292575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:36.292627 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.292649 (+ 22us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 02:57:36.292765 (+ 116us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 02:57:36.292766 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 02:57:36.292802 (+ 36us) client_negotiation.cc:240] Negotiation successful
0924 02:57:36.292840 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 02:57:36.301625 1502 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:36.301724 1502 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:36.302366 1502 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9f9ea3e78e06423080db8773cb4efcb8: Sending DeleteTablet for 3 replicas of tablet 8c235ac8df554784a382f023fd26f7d4
I20260924 02:57:36.302608 1641 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c235ac8df554784a382f023fd26f7d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35870
I20260924 02:57:36.302664 1781 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c235ac8df554784a382f023fd26f7d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42136
I20260924 02:57:36.302706 2030 tablet_replica.cc:330] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6: stopping tablet replica
I20260924 02:57:36.302749 2030 raft_consensus.cc:2229] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [term 1 LEADER]: Raft consensus shutting down.
I20260924 02:57:36.302834 2030 raft_consensus.cc:2258] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:36.302907 2032 tablet_replica.cc:330] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc: stopping tablet replica
I20260924 02:57:36.302968 2032 raft_consensus.cc:2229] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:36.303005 2032 raft_consensus.cc:2258] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:36.303187 1910 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c235ac8df554784a382f023fd26f7d4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:36 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38486
I20260924 02:57:36.303280 2031 tablet_replica.cc:330] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0: stopping tablet replica
I20260924 02:57:36.303323 2031 raft_consensus.cc:2229] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:36.303354 2031 raft_consensus.cc:2258] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:36.303706 2031 ts_tablet_manager.cc:1905] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:36.303805 2030 ts_tablet_manager.cc:1905] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:36.305477 2031 ts_tablet_manager.cc:1918] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 02:57:36.305531 2031 log.cc:1196] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/8c235ac8df554784a382f023fd26f7d4
I20260924 02:57:36.305725 2031 ts_tablet_manager.cc:1939] T 8c235ac8df554784a382f023fd26f7d4 P fdd834f4b5714d10a702fc5559b374d0: Deleting consensus metadata
I20260924 02:57:36.306005 2030 ts_tablet_manager.cc:1918] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 02:57:36.306064 2030 log.cc:1196] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-0/wal/wals/8c235ac8df554784a382f023fd26f7d4
I20260924 02:57:36.306262 2030 ts_tablet_manager.cc:1939] T 8c235ac8df554784a382f023fd26f7d4 P f632640b969c4ac4954fb8c033e101f6: Deleting consensus metadata
I20260924 02:57:36.306726 1491 catalog_manager.cc:4841] TS fdd834f4b5714d10a702fc5559b374d0 (127.0.98.3:34535): tablet 8c235ac8df554784a382f023fd26f7d4 (table test-table [id=c1af280bd1864c32a57bd9b531fbf205]) successfully deleted
I20260924 02:57:36.306821 1490 catalog_manager.cc:4841] TS f632640b969c4ac4954fb8c033e101f6 (127.0.98.1:40357): tablet 8c235ac8df554784a382f023fd26f7d4 (table test-table [id=c1af280bd1864c32a57bd9b531fbf205]) successfully deleted
I20260924 02:57:36.307049 2032 ts_tablet_manager.cc:1905] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:36.308424 2032 ts_tablet_manager.cc:1918] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 02:57:36.308475 2032 log.cc:1196] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/8c235ac8df554784a382f023fd26f7d4
I20260924 02:57:36.308665 2032 ts_tablet_manager.cc:1939] T 8c235ac8df554784a382f023fd26f7d4 P 79585bebb1c94b6ea469ae48b52655bc: Deleting consensus metadata
I20260924 02:57:36.309054 1491 catalog_manager.cc:4841] TS 79585bebb1c94b6ea469ae48b52655bc (127.0.98.2:44619): tablet 8c235ac8df554784a382f023fd26f7d4 (table test-table [id=c1af280bd1864c32a57bd9b531fbf205]) successfully deleted
Sep 24 02:57:36 dist-test-slave-sb6p krb5kdc[1459](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218656, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 02:57:36.316915 2003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.306174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35882 (local address 127.0.98.1:40357)
0924 02:57:36.306212 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.306214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.306548 (+ 334us) server_negotiation.cc:407] Connection header received
0924 02:57:36.306648 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.306648 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.306658 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.306686 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:36.308127 (+ 1441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.308852 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.310037 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.310331 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.310371 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.313738 (+ 3367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:36.313746 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:36.313746 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:36.313753 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:36.315619 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:36.315987 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:36.315990 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:36.315990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:36.316060 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:36.316256 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:36.316256 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:36.316257 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:36.316411 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:36.316527 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.316719 (+ 192us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.316799 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 02:57:36.317413 1701 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35882
I20260924 02:57:36.325350 1553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:36.318336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59778 (local address 127.0.98.62:43965)
0924 02:57:36.318377 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:36.318378 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:36.318565 (+ 187us) server_negotiation.cc:407] Connection header received
0924 02:57:36.318670 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:36.318671 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:36.318684 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:36.318745 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:36.319317 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.320096 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.321076 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:36.321376 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:36.321440 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:36.322735 (+ 1295us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:36.322737 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:36.322737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:36.322764 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:36.324284 (+ 1520us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:36.324645 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:36.324646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:36.324646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:36.324702 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:36.324870 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:36.324871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:36.324871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:36.324991 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:36.325082 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:36.325182 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 02:57:36.325256 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 02:57:36.325642 1502 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:59778
I20260924 02:57:36.326375 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 1576
I20260924 02:57:36.332196 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 1716
I20260924 02:57:36.337685 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 1856
I20260924 02:57:36.341733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 1472
2026-09-24T02:57:36Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (1982 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:57:36 dist-test-slave-sb6p krb5kdc[2068](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:57:36 dist-test-slave-sb6p krb5kdc[2068](info): set up 2 sockets
Sep 24 02:57:36 dist-test-slave-sb6p krb5kdc[2068](info): commencing operation
krb5kdc: starting...
W20260924 02:57:37.289934 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:57:37 dist-test-slave-sb6p krb5kdc[2068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:57:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:57:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 02:57:37.397580 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46545
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46545
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:37.475558 2081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:37.475759 2081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:37.475780 2081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:37.476905 2081 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:37.476948 2081 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 02:57:37.476960 2081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:37.476971 2081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:37.476994 2081 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:37.477018 2081 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:57:37.478490 2081 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46545
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46545
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:37.478726 2081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:37.478967 2081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:37.483459 2086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:37.483492 2089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:37.483493 2087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:37.483767 2081 server_base.cc:1034] running on GCE node
I20260924 02:57:37.483947 2081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:37.484206 2081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:37.485334 2081 hybrid_clock.cc:648] HybridClock initialized: now 1790218657485324 us; error 26 us; skew 500 ppm
Sep 24 02:57:37 dist-test-slave-sb6p krb5kdc[2068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:37.487636 2081 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:57:37.488049 2081 webserver.cc:460] Webserver started at http://127.0.98.62:40525/ using document root <none> and password file <none>
I20260924 02:57:37.488258 2081 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:37.488305 2081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:37.488425 2081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:37.489444 2081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "3286545749e34697a142d5d482720306"
format_stamp: "Formatted at 2026-09-24 02:57:37 on dist-test-slave-sb6p"
server_key: "530274b6ed383df7998fb123261c0e4c"
server_key_iv: "639759e1c53613717d5610e95d76b77f"
server_key_version: "encryptionkey@0"
I20260924 02:57:37.489758 2081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "3286545749e34697a142d5d482720306"
format_stamp: "Formatted at 2026-09-24 02:57:37 on dist-test-slave-sb6p"
server_key: "530274b6ed383df7998fb123261c0e4c"
server_key_iv: "639759e1c53613717d5610e95d76b77f"
server_key_version: "encryptionkey@0"
I20260924 02:57:37.490959 2081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 02:57:37.491571 2096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:37.491711 2081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:37.491787 2081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "3286545749e34697a142d5d482720306"
format_stamp: "Formatted at 2026-09-24 02:57:37 on dist-test-slave-sb6p"
server_key: "530274b6ed383df7998fb123261c0e4c"
server_key_iv: "639759e1c53613717d5610e95d76b77f"
server_key_version: "encryptionkey@0"
I20260924 02:57:37.491842 2081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:37.503185 2081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:37.505232 2081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:37.505368 2081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:37.509828 2081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46545
I20260924 02:57:37.509872 2158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46545 every 8 connection(s)
I20260924 02:57:37.510178 2081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:57:37.510799 2159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:37.511658 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 2081
I20260924 02:57:37.511734 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:57:37.511880 392 external_mini_cluster.cc:1428] Setting key 79285e9cc71217ddb3a59b090c362466
I20260924 02:57:37.513226 2159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306: Bootstrap starting.
I20260924 02:57:37.513779 2159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:37.514029 2159 log.cc:824] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:37.514710 2159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306: No bootstrap required, opened a new log
I20260924 02:57:37.516404 2159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3286545749e34697a142d5d482720306" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46545 } }
I20260924 02:57:37.516536 2159 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:37.516567 2159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3286545749e34697a142d5d482720306, State: Initialized, Role: FOLLOWER
I20260924 02:57:37.516700 2159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3286545749e34697a142d5d482720306" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46545 } }
I20260924 02:57:37.516772 2159 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:57:37.516803 2159 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:57:37.516850 2159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:37.517372 2159 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3286545749e34697a142d5d482720306" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46545 } }
I20260924 02:57:37.517463 2159 leader_election.cc:302] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3286545749e34697a142d5d482720306; no voters:
I20260924 02:57:37.517611 2159 leader_election.cc:288] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:57:37.517649 2164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:37.517819 2164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [term 1 LEADER]: Becoming Leader. State: Replica: 3286545749e34697a142d5d482720306, State: Running, Role: LEADER
I20260924 02:57:37.517854 2159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:37.517975 2164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3286545749e34697a142d5d482720306" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46545 } }
Sep 24 02:57:37 dist-test-slave-sb6p krb5kdc[2068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:37.518395 2166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3286545749e34697a142d5d482720306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3286545749e34697a142d5d482720306" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46545 } } }
I20260924 02:57:37.518471 2166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:37.518726 2173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:37.519339 2165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3286545749e34697a142d5d482720306. Latest consensus state: current_term: 1 leader_uuid: "3286545749e34697a142d5d482720306" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3286545749e34697a142d5d482720306" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46545 } } }
I20260924 02:57:37.519444 2165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:37.519738 2173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:57:37.521486 2173 catalog_manager.cc:1347] Generated new cluster ID: 8d54af48c3b54df395ee87868769be6c
I20260924 02:57:37.521536 2173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:57:37.522593 2162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.512339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38040 (local address 127.0.98.62:46545)
0924 02:57:37.512497 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:37.512506 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:37.512530 (+ 24us) server_negotiation.cc:407] Connection header received
0924 02:57:37.513047 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:37.513054 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:37.513099 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:37.513244 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:37.513819 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.514971 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.515735 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.516158 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.516261 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.518665 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:37.518670 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:37.518670 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:37.518681 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:37.520698 (+ 2017us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:37.521101 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:37.521103 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:37.521103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:37.521195 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:37.521385 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:37.521386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:37.521387 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:37.521715 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:37.521852 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:37.522245 (+ 393us) server_negotiation.cc:299] Negotiation successful
0924 02:57:37.522371 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":75,"threads_started":1}
I20260924 02:57:37.529873 2173 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:57:37.530342 2173 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:37.537151 2173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306: Generated new TSK 0
I20260924 02:57:37.537320 2173 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 02:57:37.559581 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46545
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:37.636622 2185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:37.636823 2185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:37.636847 2185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:37.637984 2185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:37.638024 2185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:37.638043 2185 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 02:57:37.638054 2185 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 02:57:37.638070 2185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:57:37.639592 2185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:37.639849 2185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:37.640096 2185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:37.644822 2190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:37.644872 2185 server_base.cc:1034] running on GCE node
W20260924 02:57:37.644801 2191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:37.644806 2193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:37.645169 2185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:37.645396 2185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:37.646548 2185 hybrid_clock.cc:648] HybridClock initialized: now 1790218657646526 us; error 37 us; skew 500 ppm
Sep 24 02:57:37 dist-test-slave-sb6p krb5kdc[2068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:37.648799 2185 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 02:57:37.649192 2185 webserver.cc:460] Webserver started at http://127.0.98.1:43767/ using document root <none> and password file <none>
I20260924 02:57:37.649384 2185 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:37.649428 2185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:37.649544 2185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:37.650556 2185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "dab79c7b14e14289a287475db0dd24ac"
format_stamp: "Formatted at 2026-09-24 02:57:37 on dist-test-slave-sb6p"
server_key: "cbdf1ef3d96c2df9cbc1a9d93ebdb94a"
server_key_iv: "0bc9d6a6c5a0d4ece6dc570177be8c51"
server_key_version: "encryptionkey@0"
I20260924 02:57:37.650893 2185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dab79c7b14e14289a287475db0dd24ac"
format_stamp: "Formatted at 2026-09-24 02:57:37 on dist-test-slave-sb6p"
server_key: "cbdf1ef3d96c2df9cbc1a9d93ebdb94a"
server_key_iv: "0bc9d6a6c5a0d4ece6dc570177be8c51"
server_key_version: "encryptionkey@0"
I20260924 02:57:37.652069 2185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:37.652755 2200 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:37.652894 2185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:37.652972 2185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "dab79c7b14e14289a287475db0dd24ac"
format_stamp: "Formatted at 2026-09-24 02:57:37 on dist-test-slave-sb6p"
server_key: "cbdf1ef3d96c2df9cbc1a9d93ebdb94a"
server_key_iv: "0bc9d6a6c5a0d4ece6dc570177be8c51"
server_key_version: "encryptionkey@0"
I20260924 02:57:37.653035 2185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:37.664788 2185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:37.666400 2185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:37.666589 2185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:37.667059 2185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:37.667094 2185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:37.667127 2185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:37.667146 2185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:37.673098 2185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:40985
I20260924 02:57:37.673220 2315 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:40985 every 8 connection(s)
I20260924 02:57:37.673478 2185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:57:37.673606 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 2185
I20260924 02:57:37.673699 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
Sep 24 02:57:37 dist-test-slave-sb6p krb5kdc[2068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:37.673884 392 external_mini_cluster.cc:1428] Setting key e1f534d9f34607d3e1eb83f314979360
I20260924 02:57:37.676702 2162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.667561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59159 (local address 127.0.98.62:46545)
0924 02:57:37.667600 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:37.667602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:37.667838 (+ 236us) server_negotiation.cc:407] Connection header received
0924 02:57:37.668387 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:37.668389 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:37.668402 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:37.668472 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:37.669285 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.670026 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.671485 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.671800 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.671877 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.674220 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:37.674222 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:37.674223 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:37.674231 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:37.675473 (+ 1242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:37.675907 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:37.675908 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:37.675908 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:37.675972 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:37.676172 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:37.676172 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:37.676173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:37.676294 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:37.676355 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:37.676564 (+ 209us) server_negotiation.cc:299] Negotiation successful
0924 02:57:37.676638 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 02:57:37.676828 2209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.667565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46545 (local address 127.0.98.1:59159)
0924 02:57:37.667727 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:37.667760 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:37.668258 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:37.668557 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:37.668562 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:37.668899 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:37.669174 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.669181 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.670114 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.670115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.671397 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.671400 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.671884 (+ 484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.671885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.672020 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.672519 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:37.672535 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:37.674135 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:37.675551 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:37.675558 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:37.675560 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:37.675841 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:37.676028 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:37.676029 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:37.676029 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:37.676121 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:37.676346 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:37.676349 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:37.676456 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 02:57:37.676529 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 02:57:37.676659 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":92,"threads_started":1}
I20260924 02:57:37.677270 2316 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46545
I20260924 02:57:37.677349 2316 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:37.677518 2316 heartbeater.cc:511] Master 127.0.98.62:46545 requested a full tablet report, sending...
I20260924 02:57:37.678429 2112 ts_manager.cc:194] Registered new tserver with Master: dab79c7b14e14289a287475db0dd24ac (127.0.98.1:40985)
I20260924 02:57:37.679345 2112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59159
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 02:57:37.683220 2162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.678189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39829 (local address 127.0.98.62:46545)
0924 02:57:37.678225 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:37.678226 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:37.678340 (+ 114us) server_negotiation.cc:407] Connection header received
0924 02:57:37.678381 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:37.678382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:37.678391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:37.678435 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:37.678762 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.679231 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.679943 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.680200 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.680258 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.681051 (+ 793us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:37.681052 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:37.681053 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:37.681058 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:37.682262 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:37.682549 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:37.682550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:37.682551 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:37.682598 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:37.682765 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:37.682766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:37.682767 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:37.682866 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:37.682918 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:37.683099 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 02:57:37.683163 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 02:57:37.683224 2323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.678159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46545 (local address 127.0.98.1:39829)
0924 02:57:37.678273 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:37.678283 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:37.678335 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:37.678436 (+ 101us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:37.678437 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:37.678631 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:37.678697 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.678698 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.679313 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.679314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.679874 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.679876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.680258 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.680259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.680320 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.680691 (+ 371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:37.680695 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:37.680977 (+ 282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:37.682326 (+ 1349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:37.682327 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:37.682328 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:37.682478 (+ 150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:37.682654 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:37.682655 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:37.682655 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:37.682709 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:37.682916 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:37.682917 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:37.683001 (+ 84us) client_negotiation.cc:769] Sending connection context
0924 02:57:37.683065 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 02:57:37.683153 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":68,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 02:57:37.697924 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46545
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:37.775524 2325 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:37.775748 2325 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:37.775775 2325 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:37.776957 2325 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:37.777004 2325 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:37.777025 2325 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 02:57:37.777037 2325 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 02:57:37.777055 2325 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:57:37.778612 2325 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2325
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:37.778823 2325 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:37.779008 2325 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:37.783776 2330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:37.783804 2325 server_base.cc:1034] running on GCE node
W20260924 02:57:37.783905 2333 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:37.783991 2331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:37.784230 2325 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:37.784430 2325 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:37.785555 2325 hybrid_clock.cc:648] HybridClock initialized: now 1790218657785533 us; error 39 us; skew 500 ppm
Sep 24 02:57:37 dist-test-slave-sb6p krb5kdc[2068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:37.787871 2325 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 02:57:37.788319 2325 webserver.cc:460] Webserver started at http://127.0.98.2:44217/ using document root <none> and password file <none>
I20260924 02:57:37.788530 2325 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:37.788575 2325 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:37.788676 2325 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:37.789952 2325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "83f06ca24b9a4a30a6fb7f9132678d46"
format_stamp: "Formatted at 2026-09-24 02:57:37 on dist-test-slave-sb6p"
server_key: "dde00321c6286b57ed5fe232cbe87e62"
server_key_iv: "ff78a67e947d47fe622d5e20499dc549"
server_key_version: "encryptionkey@0"
I20260924 02:57:37.790335 2325 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "83f06ca24b9a4a30a6fb7f9132678d46"
format_stamp: "Formatted at 2026-09-24 02:57:37 on dist-test-slave-sb6p"
server_key: "dde00321c6286b57ed5fe232cbe87e62"
server_key_iv: "ff78a67e947d47fe622d5e20499dc549"
server_key_version: "encryptionkey@0"
I20260924 02:57:37.791903 2325 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 02:57:37.792887 2340 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:37.793037 2325 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:37.793131 2325 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "83f06ca24b9a4a30a6fb7f9132678d46"
format_stamp: "Formatted at 2026-09-24 02:57:37 on dist-test-slave-sb6p"
server_key: "dde00321c6286b57ed5fe232cbe87e62"
server_key_iv: "ff78a67e947d47fe622d5e20499dc549"
server_key_version: "encryptionkey@0"
I20260924 02:57:37.793212 2325 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:37.802722 2325 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:37.804284 2325 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:37.804386 2325 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:37.804773 2325 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:37.804807 2325 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:37.804827 2325 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:37.804839 2325 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 02:57:37 dist-test-slave-sb6p krb5kdc[2068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:37.811280 2325 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:35033
I20260924 02:57:37.811595 2325 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 02:57:37.811564 2455 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:35033 every 8 connection(s)
I20260924 02:57:37.812021 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 2325
I20260924 02:57:37.812105 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:57:37.812261 392 external_mini_cluster.cc:1428] Setting key f7ca290bec02417dc775c818e1c25448
I20260924 02:57:37.814205 2162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.805198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44663 (local address 127.0.98.62:46545)
0924 02:57:37.805261 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:37.805264 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:37.805410 (+ 146us) server_negotiation.cc:407] Connection header received
0924 02:57:37.805868 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:37.805870 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:37.805883 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:37.805957 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:37.806747 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.807279 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.808283 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.808571 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.808635 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.811141 (+ 2506us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:37.811142 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:37.811142 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:37.811149 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:37.812950 (+ 1801us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:37.813392 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:37.813393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:37.813393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:37.813438 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:37.813637 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:37.813638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:37.813638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:37.813727 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:37.813768 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:37.814038 (+ 270us) server_negotiation.cc:299] Negotiation successful
0924 02:57:37.814106 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 02:57:37.814399 2349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.805141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46545 (local address 127.0.98.2:44663)
0924 02:57:37.805294 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:37.805332 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:37.805786 (+ 454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:37.806037 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:37.806040 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:37.806345 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:37.806653 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.806660 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.807356 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.807357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.808207 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.808209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.808688 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.808690 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.808851 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.809362 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:37.809371 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:37.811036 (+ 1665us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:37.813017 (+ 1981us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:37.813023 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:37.813025 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:37.813327 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:37.813476 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:37.813476 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:37.813477 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:37.813590 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:37.813778 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:37.813782 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:37.813924 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 02:57:37.814025 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 02:57:37.814187 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":90,"threads_started":1}
I20260924 02:57:37.814790 2456 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46545
I20260924 02:57:37.814862 2456 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:37.815022 2456 heartbeater.cc:511] Master 127.0.98.62:46545 requested a full tablet report, sending...
I20260924 02:57:37.815328 2113 ts_manager.cc:194] Registered new tserver with Master: 83f06ca24b9a4a30a6fb7f9132678d46 (127.0.98.2:35033)
I20260924 02:57:37.815793 2113 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:44663
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 02:57:37.821110 2162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.815918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58011 (local address 127.0.98.62:46545)
0924 02:57:37.815975 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:37.815976 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:37.815984 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:57:37.815999 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:37.816000 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:37.816008 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:37.816053 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:37.816491 (+ 438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.816996 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.817715 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.817994 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.818056 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.818934 (+ 878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:37.818935 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:37.818936 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:37.818942 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:37.820241 (+ 1299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:37.820555 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:37.820556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:37.820556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:37.820608 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:37.820750 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:37.820750 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:37.820751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:37.820865 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:37.820908 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:37.821004 (+ 96us) server_negotiation.cc:299] Negotiation successful
0924 02:57:37.821053 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 02:57:37.821184 2463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.815718 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46545 (local address 127.0.98.2:58011)
0924 02:57:37.815841 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:37.815852 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:37.815901 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:37.816113 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:37.816115 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:37.816321 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:37.816392 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.816394 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.817085 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.817086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.817646 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.817648 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.818074 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.818075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.818132 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.818546 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:37.818551 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:37.818856 (+ 305us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:37.820315 (+ 1459us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:37.820315 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:37.820316 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:37.820490 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:37.820671 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:37.820672 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:37.820672 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:37.820717 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:37.820904 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:37.820904 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:37.820933 (+ 29us) client_negotiation.cc:769] Sending connection context
0924 02:57:37.821031 (+ 98us) client_negotiation.cc:240] Negotiation successful
0924 02:57:37.821102 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 02:57:37.835470 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46545
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43547
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:37.912285 2465 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:37.912474 2465 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:37.912496 2465 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:37.913664 2465 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:37.913704 2465 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:37.913723 2465 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260924 02:57:37.913744 2465 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 02:57:37.913771 2465 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:57:37.915305 2465 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43547
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46545
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2465
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:37.915517 2465 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:37.915714 2465 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:37.920291 2471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:37.920293 2473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:37.920282 2470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:37.920396 2465 server_base.cc:1034] running on GCE node
I20260924 02:57:37.920604 2465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:37.920802 2465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:37.921927 2465 hybrid_clock.cc:648] HybridClock initialized: now 1790218657921914 us; error 27 us; skew 500 ppm
Sep 24 02:57:37 dist-test-slave-sb6p krb5kdc[2068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:37.924117 2465 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 02:57:37.924515 2465 webserver.cc:460] Webserver started at http://127.0.98.3:34313/ using document root <none> and password file <none>
I20260924 02:57:37.924703 2465 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:37.924749 2465 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:37.924865 2465 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:37.925830 2465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "23eecc9817184a5ab3ad91ecc028abd5"
format_stamp: "Formatted at 2026-09-24 02:57:37 on dist-test-slave-sb6p"
server_key: "f0c7faeb1cb4f44e1af34fd25a2286eb"
server_key_iv: "641874fecf96906f34acd152eb412b2b"
server_key_version: "encryptionkey@0"
I20260924 02:57:37.926151 2465 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "23eecc9817184a5ab3ad91ecc028abd5"
format_stamp: "Formatted at 2026-09-24 02:57:37 on dist-test-slave-sb6p"
server_key: "f0c7faeb1cb4f44e1af34fd25a2286eb"
server_key_iv: "641874fecf96906f34acd152eb412b2b"
server_key_version: "encryptionkey@0"
I20260924 02:57:37.927330 2465 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 02:57:37.927938 2480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:37.928072 2465 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:37.928146 2465 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "23eecc9817184a5ab3ad91ecc028abd5"
format_stamp: "Formatted at 2026-09-24 02:57:37 on dist-test-slave-sb6p"
server_key: "f0c7faeb1cb4f44e1af34fd25a2286eb"
server_key_iv: "641874fecf96906f34acd152eb412b2b"
server_key_version: "encryptionkey@0"
I20260924 02:57:37.928201 2465 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:37.945474 2465 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:37.947196 2465 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:37.947327 2465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:37.947810 2465 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:37.947847 2465 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:37.947876 2465 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:37.947892 2465 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:37.954146 2465 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46009
I20260924 02:57:37.954219 2595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46009 every 8 connection(s)
Sep 24 02:57:37 dist-test-slave-sb6p krb5kdc[2068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:37.954476 2465 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 02:57:37.957223 2162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.948219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33209 (local address 127.0.98.62:46545)
0924 02:57:37.948265 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:37.948267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:37.948489 (+ 222us) server_negotiation.cc:407] Connection header received
0924 02:57:37.948988 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:37.948990 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:37.949003 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:37.949060 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:37.949918 (+ 858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.950700 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.952168 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.952457 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.952515 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.954920 (+ 2405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:37.954921 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:37.954921 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:37.954929 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:37.956060 (+ 1131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:37.956385 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:37.956386 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:37.956386 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:37.956436 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:37.956618 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:37.956618 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:37.956618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:37.956726 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:37.956774 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:37.957100 (+ 326us) server_negotiation.cc:299] Negotiation successful
0924 02:57:37.957159 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 02:57:37.957393 2489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.948229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46545 (local address 127.0.98.3:33209)
0924 02:57:37.948380 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:37.948413 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:37.948897 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:37.949182 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:37.949187 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:37.949530 (+ 343us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:37.949814 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.949821 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.950807 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.950808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.952077 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.952081 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.952534 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.952536 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.952673 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.953201 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:37.953219 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:37.954839 (+ 1620us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:37.956121 (+ 1282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:37.956128 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:37.956130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:37.956326 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:37.956488 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:37.956488 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:37.956488 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:37.956571 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:37.956844 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:37.956851 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:37.956981 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 02:57:37.957081 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 02:57:37.957198 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":89,"threads_started":1}
I20260924 02:57:37.957839 2596 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46545
I20260924 02:57:37.957914 2596 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:37.958065 2596 heartbeater.cc:511] Master 127.0.98.62:46545 requested a full tablet report, sending...
I20260924 02:57:37.958407 2112 ts_manager.cc:194] Registered new tserver with Master: 23eecc9817184a5ab3ad91ecc028abd5 (127.0.98.3:46009)
I20260924 02:57:37.958798 2112 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33209
I20260924 02:57:37.959546 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 2465
I20260924 02:57:37.959646 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:57:37.959807 392 external_mini_cluster.cc:1428] Setting key daedd0c1369ede6430d965f87008acc1
I20260924 02:57:37.960561 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 02:57:37.964332 2162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.958865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32995 (local address 127.0.98.62:46545)
0924 02:57:37.958902 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:37.958903 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:37.959161 (+ 258us) server_negotiation.cc:407] Connection header received
0924 02:57:37.959183 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:37.959183 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:37.959193 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:37.959245 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:37.959586 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.960157 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.960916 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.961199 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.961262 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.962177 (+ 915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:37.962178 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:37.962178 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:37.962185 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:37.963389 (+ 1204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:37.963716 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:37.963717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:37.963717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:37.963758 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:37.963931 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:37.963932 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:37.963932 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:37.964025 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:37.964072 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:37.964226 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 02:57:37.964276 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 02:57:37.964396 2602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.958799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46545 (local address 127.0.98.3:32995)
0924 02:57:37.959093 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:37.959102 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:37.959150 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:37.959255 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:37.959256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:37.959438 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:37.959511 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.959513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.960228 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.960229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.960847 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.960850 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.961285 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.961285 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.961347 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.961746 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:37.961751 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:37.962044 (+ 293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:37.963463 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:37.963464 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:37.963464 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:37.963650 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:37.963814 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:37.963815 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:37.963815 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:37.963876 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:37.964076 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:37.964077 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:37.964119 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 02:57:37.964205 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 02:57:37.964295 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":175,"threads_started":1}
Sep 24 02:57:37 dist-test-slave-sb6p krb5kdc[2068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 02:57:37 dist-test-slave-sb6p krb5kdc[2068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218657, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:37.980741 2162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.974986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38044 (local address 127.0.98.62:46545)
0924 02:57:37.975037 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:37.975038 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:37.975062 (+ 24us) server_negotiation.cc:407] Connection header received
0924 02:57:37.975079 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:37.975080 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:37.975087 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:37.975121 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:37.975609 (+ 488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.976068 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.976736 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.976903 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.976944 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.978852 (+ 1908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:37.978854 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:37.978854 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:37.978860 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:37.979837 (+ 977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:37.980118 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:37.980119 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:37.980119 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:37.980164 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:37.980315 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:37.980316 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:37.980316 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:37.980409 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:37.980487 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:37.980581 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 02:57:37.980656 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 02:57:37.981986 2112 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 02:57:37.983554 2112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:57:37.989241 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.985744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37238 (local address 127.0.98.3:46009)
0924 02:57:37.985913 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:37.985917 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:37.985925 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:57:37.985957 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:37.985959 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:37.985971 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:37.986046 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:37.986180 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.986967 (+ 787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.988297 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.988875 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.988937 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.988992 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:37.989111 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 02:57:37.989147 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":82,"threads_started":1}
I20260924 02:57:37.989248 2611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.985364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40985 (local address 127.0.0.1:42434)
0924 02:57:37.985491 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:37.985530 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:37.985592 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:37.986036 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:37.986036 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:37.986038 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:37.986073 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.986074 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.987351 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.987352 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.988125 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.988127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.988986 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.988987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.989049 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.989053 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:57:37.989080 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 02:57:37.989163 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":77,"threads_started":1}
I20260924 02:57:37.989243 2610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.985369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46009 (local address 127.0.0.1:37238)
0924 02:57:37.985766 (+ 397us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:37.985774 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:37.985812 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:37.986049 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:37.986049 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:37.986052 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:37.986094 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.986096 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.987038 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.987039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.988208 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.988211 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.988945 (+ 734us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.988946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.989018 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.989031 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 02:57:37.989092 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 02:57:37.989164 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":387,"mutex_wait_us":28,"thread_start_us":87,"threads_started":1}
I20260924 02:57:37.989388 2612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.985364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35033 (local address 127.0.0.1:43932)
0924 02:57:37.985710 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:37.985718 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:37.985764 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:37.985932 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:37.985936 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:37.985937 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:37.986012 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.986014 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.987309 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.987310 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.988385 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:37.988387 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.989214 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.989215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:37.989271 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.989274 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:37.989301 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 02:57:37.989335 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":330,"mutex_wait_us":5,"thread_start_us":43,"threads_started":1}
I20260924 02:57:37.989697 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.985583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43932 (local address 127.0.98.2:35033)
0924 02:57:37.985723 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:37.985728 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:37.985758 (+ 30us) server_negotiation.cc:407] Connection header received
0924 02:57:37.985830 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:37.985832 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:37.985847 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:37.985930 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:37.986094 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.987232 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.988456 (+ 1224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.989131 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.989463 (+ 332us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.989528 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:37.989553 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 02:57:37.989588 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
I20260924 02:57:37.989858 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:37.985601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.0.98.1:40985)
0924 02:57:37.985870 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:37.985875 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:37.985883 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:57:37.985917 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:37.985919 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:37.985933 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:37.986159 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:37.986214 (+ 55us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.987260 (+ 1046us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.988217 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:37.988891 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:37.989649 (+ 758us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:37.989714 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:37.989739 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 02:57:37.989778 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":241,"thread_start_us":82,"threads_started":1}
I20260924 02:57:37.990069 2527 tablet_service.cc:1467] Processing CreateTablet for tablet 84fb04a777624facadc33a0205dc02a5 (DEFAULT_TABLE table=test-table [id=fef31da88b734a059472923744eb989f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:37.990427 2527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84fb04a777624facadc33a0205dc02a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:37.990350 2390 tablet_service.cc:1467] Processing CreateTablet for tablet 84fb04a777624facadc33a0205dc02a5 (DEFAULT_TABLE table=test-table [id=fef31da88b734a059472923744eb989f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:37.990372 2250 tablet_service.cc:1467] Processing CreateTablet for tablet 84fb04a777624facadc33a0205dc02a5 (DEFAULT_TABLE table=test-table [id=fef31da88b734a059472923744eb989f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:37.990680 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84fb04a777624facadc33a0205dc02a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:37.990684 2250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84fb04a777624facadc33a0205dc02a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:37.992791 2616 tablet_bootstrap.cc:492] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5: Bootstrap starting.
I20260924 02:57:37.993752 2616 tablet_bootstrap.cc:654] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:37.994099 2616 log.cc:824] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:37.994153 2617 tablet_bootstrap.cc:492] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac: Bootstrap starting.
I20260924 02:57:37.994603 2618 tablet_bootstrap.cc:492] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46: Bootstrap starting.
I20260924 02:57:37.994750 2617 tablet_bootstrap.cc:654] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:37.994805 2616 tablet_bootstrap.cc:492] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5: No bootstrap required, opened a new log
I20260924 02:57:37.994863 2616 ts_tablet_manager.cc:1397] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:37.995127 2618 tablet_bootstrap.cc:654] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:37.995143 2617 log.cc:824] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:37.995445 2618 log.cc:824] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:37.995832 2617 tablet_bootstrap.cc:492] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac: No bootstrap required, opened a new log
I20260924 02:57:37.995896 2617 ts_tablet_manager.cc:1397] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:37.996059 2618 tablet_bootstrap.cc:492] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46: No bootstrap required, opened a new log
I20260924 02:57:37.996112 2618 ts_tablet_manager.cc:1397] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:37.996726 2616 raft_consensus.cc:348] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } }
I20260924 02:57:37.996884 2616 raft_consensus.cc:374] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:37.996925 2616 raft_consensus.cc:729] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eecc9817184a5ab3ad91ecc028abd5, State: Initialized, Role: FOLLOWER
I20260924 02:57:37.997049 2616 consensus_queue.cc:260] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } }
I20260924 02:57:37.997295 2616 ts_tablet_manager.cc:1428] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 02:57:37.997432 2596 heartbeater.cc:503] Master 127.0.98.62:46545 was elected leader, sending a full tablet report...
I20260924 02:57:37.997783 2617 raft_consensus.cc:348] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } }
I20260924 02:57:37.997927 2617 raft_consensus.cc:374] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:37.997956 2617 raft_consensus.cc:729] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab79c7b14e14289a287475db0dd24ac, State: Initialized, Role: FOLLOWER
I20260924 02:57:37.997937 2618 raft_consensus.cc:348] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } }
I20260924 02:57:37.998047 2618 raft_consensus.cc:374] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:37.998076 2618 raft_consensus.cc:729] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f06ca24b9a4a30a6fb7f9132678d46, State: Initialized, Role: FOLLOWER
I20260924 02:57:37.998059 2617 consensus_queue.cc:260] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } }
I20260924 02:57:37.998178 2618 consensus_queue.cc:260] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } }
I20260924 02:57:37.998306 2617 ts_tablet_manager.cc:1428] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 02:57:37.998411 2316 heartbeater.cc:503] Master 127.0.98.62:46545 was elected leader, sending a full tablet report...
I20260924 02:57:37.998409 2618 ts_tablet_manager.cc:1428] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 02:57:37.998448 2456 heartbeater.cc:503] Master 127.0.98.62:46545 was elected leader, sending a full tablet report...
W20260924 02:57:38.062301 2457 tablet.cc:2369] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 02:57:38.152727 2622 raft_consensus.cc:482] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:57:38.152848 2622 raft_consensus.cc:504] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } }
I20260924 02:57:38.153148 2622 leader_election.cc:288] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dab79c7b14e14289a287475db0dd24ac (127.0.98.1:40985), 83f06ca24b9a4a30a6fb7f9132678d46 (127.0.98.2:35033)
I20260924 02:57:38.155797 2613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:38.153494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42747 (local address 127.0.98.2:35033)
0924 02:57:38.153542 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:38.153543 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:38.153550 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:57:38.153572 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:38.153573 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:38.153582 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:38.153635 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:38.153760 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.154227 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.155155 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.155509 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.155566 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:38.155576 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:38.155700 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 02:57:38.155733 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 02:57:38.155798 2489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:38.153340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35033 (local address 127.0.98.3:42747)
0924 02:57:38.153404 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:38.153415 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:38.153457 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:38.153638 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:38.153639 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:38.153640 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:38.153706 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:38.153708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.154299 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.154300 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:38.155080 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:38.155082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.155575 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.155576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:38.155644 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:38.155647 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:38.155690 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 02:57:38.155727 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":1}
I20260924 02:57:38.155865 2614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:38.153430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44027 (local address 127.0.98.1:40985)
0924 02:57:38.153492 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:38.153494 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:38.153617 (+ 123us) server_negotiation.cc:407] Connection header received
0924 02:57:38.153649 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:38.153651 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:38.153662 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:38.153708 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:38.153838 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.154286 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.155187 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.155576 (+ 389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.155640 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:38.155649 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:38.155769 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 02:57:38.155801 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 02:57:38.155901 2625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:38.153348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40985 (local address 127.0.98.3:44027)
0924 02:57:38.153532 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:38.153542 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:38.153605 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:38.153714 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:38.153715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:38.153716 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:38.153783 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:38.153784 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.154352 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.154353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:38.155104 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:38.155106 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.155641 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.155642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:38.155713 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:38.155717 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:57:38.155760 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 02:57:38.155794 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164,"mutex_wait_us":19,"thread_start_us":99,"threads_started":1}
I20260924 02:57:38.156170 2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fb04a777624facadc33a0205dc02a5" candidate_uuid: "23eecc9817184a5ab3ad91ecc028abd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" is_pre_election: true
I20260924 02:57:38.156229 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fb04a777624facadc33a0205dc02a5" candidate_uuid: "23eecc9817184a5ab3ad91ecc028abd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79c7b14e14289a287475db0dd24ac" is_pre_election: true
I20260924 02:57:38.156304 2410 raft_consensus.cc:2454] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eecc9817184a5ab3ad91ecc028abd5 in term 0.
I20260924 02:57:38.156379 2270 raft_consensus.cc:2454] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eecc9817184a5ab3ad91ecc028abd5 in term 0.
I20260924 02:57:38.156502 2482 leader_election.cc:302] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eecc9817184a5ab3ad91ecc028abd5, 83f06ca24b9a4a30a6fb7f9132678d46; no voters:
I20260924 02:57:38.156668 2622 raft_consensus.cc:2781] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:57:38.156705 2622 raft_consensus.cc:482] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:57:38.156720 2622 raft_consensus.cc:3037] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:38.157357 2622 raft_consensus.cc:504] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } }
I20260924 02:57:38.157466 2622 leader_election.cc:288] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [CANDIDATE]: Term 1 election: Requested vote from peers dab79c7b14e14289a287475db0dd24ac (127.0.98.1:40985), 83f06ca24b9a4a30a6fb7f9132678d46 (127.0.98.2:35033)
I20260924 02:57:38.157605 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fb04a777624facadc33a0205dc02a5" candidate_uuid: "23eecc9817184a5ab3ad91ecc028abd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79c7b14e14289a287475db0dd24ac"
I20260924 02:57:38.157600 2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84fb04a777624facadc33a0205dc02a5" candidate_uuid: "23eecc9817184a5ab3ad91ecc028abd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f06ca24b9a4a30a6fb7f9132678d46"
I20260924 02:57:38.157696 2410 raft_consensus.cc:3037] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:38.157696 2270 raft_consensus.cc:3037] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:38.158483 2410 raft_consensus.cc:2454] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eecc9817184a5ab3ad91ecc028abd5 in term 1.
I20260924 02:57:38.158483 2270 raft_consensus.cc:2454] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eecc9817184a5ab3ad91ecc028abd5 in term 1.
I20260924 02:57:38.158977 2484 leader_election.cc:302] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eecc9817184a5ab3ad91ecc028abd5, dab79c7b14e14289a287475db0dd24ac; no voters:
I20260924 02:57:38.159073 2622 raft_consensus.cc:2781] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:38.159230 2622 raft_consensus.cc:686] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 LEADER]: Becoming Leader. State: Replica: 23eecc9817184a5ab3ad91ecc028abd5, State: Running, Role: LEADER
I20260924 02:57:38.159312 2622 consensus_queue.cc:237] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } }
I20260924 02:57:38.160105 2111 catalog_manager.cc:5495] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23eecc9817184a5ab3ad91ecc028abd5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "23eecc9817184a5ab3ad91ecc028abd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } health_report { overall_health: UNKNOWN } } }
W20260924 02:57:38.174388 2317 tablet.cc:2369] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 02:57:38 dist-test-slave-sb6p krb5kdc[2068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218657, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:38.193572 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:38.187175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46545 (local address 127.0.0.1:38046)
0924 02:57:38.187323 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:38.187333 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:38.187402 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:38.187523 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:38.187524 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:38.187833 (+ 309us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:38.187883 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:38.187885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.188505 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.188506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:38.189014 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:38.189015 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.189319 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.189320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:38.189369 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:38.189862 (+ 493us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:38.189867 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:38.191208 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:38.192722 (+ 1514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:38.192728 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:38.192730 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:38.192916 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:38.193060 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:38.193060 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:38.193061 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:38.193109 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:38.193288 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:38.193294 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:38.193337 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 02:57:38.193388 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 02:57:38.193476 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":89,"threads_started":1}
I20260924 02:57:38.193621 2162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:38.187232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38046 (local address 127.0.98.62:46545)
0924 02:57:38.187288 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:38.187290 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:38.187397 (+ 107us) server_negotiation.cc:407] Connection header received
0924 02:57:38.187466 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:38.187467 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:38.187475 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:38.187514 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:38.187958 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.188437 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.189079 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.189261 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.189314 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:38.191305 (+ 1991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:38.191306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:38.191307 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:38.191313 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:38.192637 (+ 1324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:38.192971 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:38.192972 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:38.192972 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:38.193017 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:38.193148 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:38.193149 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:38.193149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:38.193241 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:38.193284 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:38.193414 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 02:57:38.193517 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":31}
I20260924 02:57:38.195526 2111 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38046:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 02:57:38.195703 2111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:57:38.196964 2250 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8d96b0b54a4459ba1a541f62f931ed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0a975492891b44b3bbfa6f4daeb2f96a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 02:57:38.196972 2527 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8d96b0b54a4459ba1a541f62f931ed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0a975492891b44b3bbfa6f4daeb2f96a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 02:57:38.196987 2390 tablet_service.cc:1467] Processing CreateTablet for tablet 1d8d96b0b54a4459ba1a541f62f931ed (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0a975492891b44b3bbfa6f4daeb2f96a]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 02:57:38.197175 2527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8d96b0b54a4459ba1a541f62f931ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:38.197175 2250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8d96b0b54a4459ba1a541f62f931ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:38.197181 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1d8d96b0b54a4459ba1a541f62f931ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:38.198256 2618 tablet_bootstrap.cc:492] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46: Bootstrap starting.
I20260924 02:57:38.198552 2617 tablet_bootstrap.cc:492] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac: Bootstrap starting.
I20260924 02:57:38.198557 2616 tablet_bootstrap.cc:492] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5: Bootstrap starting.
I20260924 02:57:38.198863 2617 tablet_bootstrap.cc:654] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:38.198951 2616 tablet_bootstrap.cc:654] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:38.198982 2618 tablet_bootstrap.cc:654] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:38.199409 2617 tablet_bootstrap.cc:492] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac: No bootstrap required, opened a new log
I20260924 02:57:38.199460 2617 ts_tablet_manager.cc:1397] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 02:57:38.199612 2617 raft_consensus.cc:348] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.199687 2617 raft_consensus.cc:374] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:38.199710 2617 raft_consensus.cc:729] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab79c7b14e14289a287475db0dd24ac, State: Initialized, Role: FOLLOWER
I20260924 02:57:38.199751 2617 consensus_queue.cc:260] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.199846 2617 ts_tablet_manager.cc:1428] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:38.199880 2623 tablet_replica.cc:439] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } }
I20260924 02:57:38.199955 2623 tablet_replica.cc:442] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:38.200248 2618 tablet_bootstrap.cc:492] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46: No bootstrap required, opened a new log
I20260924 02:57:38.200295 2618 ts_tablet_manager.cc:1397] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:38.200371 2616 tablet_bootstrap.cc:492] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5: No bootstrap required, opened a new log
I20260924 02:57:38.200410 2616 ts_tablet_manager.cc:1397] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:38.200433 2618 raft_consensus.cc:348] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.200489 2618 raft_consensus.cc:374] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:38.200501 2618 raft_consensus.cc:729] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f06ca24b9a4a30a6fb7f9132678d46, State: Initialized, Role: FOLLOWER
I20260924 02:57:38.200531 2618 consensus_queue.cc:260] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.200534 2616 raft_consensus.cc:348] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.200600 2618 ts_tablet_manager.cc:1428] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:38.200615 2616 raft_consensus.cc:374] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:38.200637 2616 raft_consensus.cc:729] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eecc9817184a5ab3ad91ecc028abd5, State: Initialized, Role: FOLLOWER
I20260924 02:57:38.200677 2616 consensus_queue.cc:260] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.200747 2616 ts_tablet_manager.cc:1428] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:38.200733 2624 tablet_replica.cc:439] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } }
I20260924 02:57:38.200796 2624 tablet_replica.cc:442] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:38.200798 2622 tablet_replica.cc:439] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } }
I20260924 02:57:38.200866 2622 tablet_replica.cc:442] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 02:57:38.204990 2597 tablet.cc:2369] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 02:57:38.366138 2622 raft_consensus.cc:482] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:57:38.366263 2622 raft_consensus.cc:504] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.366425 2622 leader_election.cc:288] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dab79c7b14e14289a287475db0dd24ac (127.0.98.1:40985), 83f06ca24b9a4a30a6fb7f9132678d46 (127.0.98.2:35033)
I20260924 02:57:38.366665 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8d96b0b54a4459ba1a541f62f931ed" candidate_uuid: "23eecc9817184a5ab3ad91ecc028abd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79c7b14e14289a287475db0dd24ac" is_pre_election: true
I20260924 02:57:38.366679 2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8d96b0b54a4459ba1a541f62f931ed" candidate_uuid: "23eecc9817184a5ab3ad91ecc028abd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" is_pre_election: true
I20260924 02:57:38.366758 2410 raft_consensus.cc:2454] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eecc9817184a5ab3ad91ecc028abd5 in term 0.
I20260924 02:57:38.366762 2270 raft_consensus.cc:2454] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eecc9817184a5ab3ad91ecc028abd5 in term 0.
I20260924 02:57:38.366899 2482 leader_election.cc:302] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eecc9817184a5ab3ad91ecc028abd5, 83f06ca24b9a4a30a6fb7f9132678d46; no voters:
I20260924 02:57:38.367007 2622 raft_consensus.cc:2781] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:57:38.367050 2622 raft_consensus.cc:482] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:57:38.367074 2622 raft_consensus.cc:3037] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:38.367687 2622 raft_consensus.cc:504] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.367785 2622 leader_election.cc:288] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [CANDIDATE]: Term 1 election: Requested vote from peers dab79c7b14e14289a287475db0dd24ac (127.0.98.1:40985), 83f06ca24b9a4a30a6fb7f9132678d46 (127.0.98.2:35033)
I20260924 02:57:38.367936 2410 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8d96b0b54a4459ba1a541f62f931ed" candidate_uuid: "23eecc9817184a5ab3ad91ecc028abd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f06ca24b9a4a30a6fb7f9132678d46"
I20260924 02:57:38.367945 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1d8d96b0b54a4459ba1a541f62f931ed" candidate_uuid: "23eecc9817184a5ab3ad91ecc028abd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79c7b14e14289a287475db0dd24ac"
I20260924 02:57:38.368011 2270 raft_consensus.cc:3037] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:38.368012 2410 raft_consensus.cc:3037] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:38.368695 2410 raft_consensus.cc:2454] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eecc9817184a5ab3ad91ecc028abd5 in term 1.
I20260924 02:57:38.368693 2270 raft_consensus.cc:2454] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eecc9817184a5ab3ad91ecc028abd5 in term 1.
I20260924 02:57:38.368837 2482 leader_election.cc:302] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eecc9817184a5ab3ad91ecc028abd5, 83f06ca24b9a4a30a6fb7f9132678d46; no voters:
I20260924 02:57:38.368938 2622 raft_consensus.cc:2781] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:38.368994 2622 raft_consensus.cc:686] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 LEADER]: Becoming Leader. State: Replica: 23eecc9817184a5ab3ad91ecc028abd5, State: Running, Role: LEADER
I20260924 02:57:38.369046 2622 consensus_queue.cc:237] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.369340 2626 tablet_replica.cc:439] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5: TxnStatusTablet state changed. Reason: New leader 23eecc9817184a5ab3ad91ecc028abd5. Latest consensus state: current_term: 1 leader_uuid: "23eecc9817184a5ab3ad91ecc028abd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } }
I20260924 02:57:38.369414 2626 tablet_replica.cc:442] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5: This TxnStatusTablet replica's current role is: LEADER
I20260924 02:57:38.369673 2113 catalog_manager.cc:5495] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23eecc9817184a5ab3ad91ecc028abd5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "23eecc9817184a5ab3ad91ecc028abd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260924 02:57:38.369938 2638 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 02:57:38.370049 2638 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 02:57:38.382709 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:38.380098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.0.98.3:46009)
0924 02:57:38.380174 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:38.380176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:38.380183 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:57:38.380209 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:38.380209 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:38.380225 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:38.380279 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:38.380393 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.380851 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.381936 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.382167 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.382225 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:38.382421 (+ 196us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:38.382508 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:38.382593 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:38.382631 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 02:57:38.382652 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260924 02:57:38.382759 2633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:38.380004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46009 (local address 127.0.0.1:37244)
0924 02:57:38.380071 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:38.380083 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:38.380131 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:38.380292 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:38.380293 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:38.380295 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 02:57:38.380344 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:38.380346 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.380911 (+ 565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.380912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:38.381839 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:38.381841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.382234 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.382235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:38.382304 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:38.382321 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 02:57:38.382574 (+ 253us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 02:57:38.382581 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 02:57:38.382626 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 02:57:38.382679 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 02:57:38.384358 2410 raft_consensus.cc:1261] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 FOLLOWER]: Refusing update from remote peer 23eecc9817184a5ab3ad91ecc028abd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:38.384375 2270 raft_consensus.cc:1261] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac [term 1 FOLLOWER]: Refusing update from remote peer 23eecc9817184a5ab3ad91ecc028abd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:38.384685 2626 consensus_queue.cc:1035] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:38.384794 2622 consensus_queue.cc:1035] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:38.385866 2623 tablet_replica.cc:439] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac: TxnStatusTablet state changed. Reason: New leader 23eecc9817184a5ab3ad91ecc028abd5. Latest consensus state: current_term: 1 leader_uuid: "23eecc9817184a5ab3ad91ecc028abd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } }
I20260924 02:57:38.385876 2624 tablet_replica.cc:439] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46: TxnStatusTablet state changed. Reason: New leader 23eecc9817184a5ab3ad91ecc028abd5. Latest consensus state: current_term: 1 leader_uuid: "23eecc9817184a5ab3ad91ecc028abd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } }
I20260924 02:57:38.385950 2623 tablet_replica.cc:442] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:38.385957 2624 tablet_replica.cc:442] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:38.386669 2626 tablet_replica.cc:439] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23eecc9817184a5ab3ad91ecc028abd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } }
I20260924 02:57:38.386746 2626 tablet_replica.cc:442] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5: This TxnStatusTablet replica's current role is: LEADER
I20260924 02:57:38.386808 2641 mvcc.cc:205] Tried to move back new op lower bound from 7332735624740098048 to 7332735624679845888. Current Snapshot: MvccSnapshot[applied={T|T < 7332735624740098048}]
I20260924 02:57:38.387017 2622 tablet_replica.cc:439] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "23eecc9817184a5ab3ad91ecc028abd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } }
I20260924 02:57:38.387085 2622 tablet_replica.cc:442] T 1d8d96b0b54a4459ba1a541f62f931ed P 23eecc9817184a5ab3ad91ecc028abd5: This TxnStatusTablet replica's current role is: LEADER
I20260924 02:57:38.387601 2624 tablet_replica.cc:439] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23eecc9817184a5ab3ad91ecc028abd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } }
I20260924 02:57:38.387682 2624 tablet_replica.cc:442] T 1d8d96b0b54a4459ba1a541f62f931ed P 83f06ca24b9a4a30a6fb7f9132678d46: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:38.387874 2623 tablet_replica.cc:439] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "23eecc9817184a5ab3ad91ecc028abd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } } }
I20260924 02:57:38.387948 2623 tablet_replica.cc:442] T 1d8d96b0b54a4459ba1a541f62f931ed P dab79c7b14e14289a287475db0dd24ac: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:38.388065 2643 mvcc.cc:205] Tried to move back new op lower bound from 7332735624740098048 to 7332735624679845888. Current Snapshot: MvccSnapshot[applied={T|T < 7332735624740098048}]
I20260924 02:57:38.388175 2642 mvcc.cc:205] Tried to move back new op lower bound from 7332735624740098048 to 7332735624679845888. Current Snapshot: MvccSnapshot[applied={T|T < 7332735624740098048}]
I20260924 02:57:38.391222 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:38.389182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.0.98.3:46009)
0924 02:57:38.389223 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:38.389225 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:38.389232 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:57:38.389250 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:38.389251 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:38.389259 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:38.389301 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:38.389421 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.389860 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.390610 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.390827 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.390880 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:38.391014 (+ 134us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:38.391045 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:38.391088 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:38.391147 (+ 59us) server_negotiation.cc:299] Negotiation successful
0924 02:57:38.391162 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 02:57:38.395852 2615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:38.393501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52051 (local address 127.0.98.3:46009)
0924 02:57:38.393545 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:38.393547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:38.393553 (+ 6us) server_negotiation.cc:407] Connection header received
0924 02:57:38.393567 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:38.393567 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:38.393574 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:38.393620 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:38.393739 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.394160 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.395272 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.395445 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.395500 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:38.395663 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:38.395685 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:38.395718 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:38.395770 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 02:57:38.395786 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 02:57:38.395921 2602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:38.393310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46009 (local address 127.0.98.3:52051)
0924 02:57:38.393369 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:38.393379 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:38.393431 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:38.393629 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:38.393630 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:38.393631 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 02:57:38.393688 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:38.393690 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.394219 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.394220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:38.395202 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:38.395205 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:38.395508 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:38.395509 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:38.395570 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:38.395584 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 02:57:38.395722 (+ 138us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 02:57:38.395724 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 02:57:38.395767 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 02:57:38.395809 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31,"mutex_wait_us":2}
I20260924 02:57:38.397709 2410 raft_consensus.cc:1261] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 FOLLOWER]: Refusing update from remote peer 23eecc9817184a5ab3ad91ecc028abd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:38.397710 2270 raft_consensus.cc:1261] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac [term 1 FOLLOWER]: Refusing update from remote peer 23eecc9817184a5ab3ad91ecc028abd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:38.397900 2652 consensus_queue.cc:1035] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:38.397987 2622 consensus_queue.cc:1035] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:38.412997 2111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:38.413110 2111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:38.413866 2111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306: Sending DeleteTablet for 3 replicas of tablet 84fb04a777624facadc33a0205dc02a5
I20260924 02:57:38.414151 2527 tablet_service.cc:1514] Processing DeleteTablet for tablet 84fb04a777624facadc33a0205dc02a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37238
I20260924 02:57:38.414361 2390 tablet_service.cc:1514] Processing DeleteTablet for tablet 84fb04a777624facadc33a0205dc02a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43932
I20260924 02:57:38.414348 2111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38044:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 02:57:38.414475 2111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:57:38.414796 2250 tablet_service.cc:1514] Processing DeleteTablet for tablet 84fb04a777624facadc33a0205dc02a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:38 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42434
I20260924 02:57:38.415203 2658 tablet_replica.cc:330] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac: stopping tablet replica
I20260924 02:57:38.415329 2658 raft_consensus.cc:2229] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:38.415400 2658 raft_consensus.cc:2258] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:38.415882 2658 ts_tablet_manager.cc:1905] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:38.416363 2250 tablet_service.cc:1467] Processing CreateTablet for tablet 4bdd4b0614004b5199cd85cdd403ec75 (DEFAULT_TABLE table=test-table [id=49ced53625b944a58f20168d8680c592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:38.416476 2250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bdd4b0614004b5199cd85cdd403ec75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:38.417665 2658 ts_tablet_manager.cc:1918] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 02:57:38.417793 2617 tablet_bootstrap.cc:492] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac: Bootstrap starting.
I20260924 02:57:38.417783 2658 log.cc:1196] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/wal/wals/84fb04a777624facadc33a0205dc02a5
I20260924 02:57:38.418112 2658 ts_tablet_manager.cc:1939] T 84fb04a777624facadc33a0205dc02a5 P dab79c7b14e14289a287475db0dd24ac: Deleting consensus metadata
I20260924 02:57:38.418207 2617 tablet_bootstrap.cc:654] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:38.418697 2100 catalog_manager.cc:4841] TS dab79c7b14e14289a287475db0dd24ac (127.0.98.1:40985): tablet 84fb04a777624facadc33a0205dc02a5 (table test-table [id=fef31da88b734a059472923744eb989f]) successfully deleted
I20260924 02:57:38.418920 2390 tablet_service.cc:1467] Processing CreateTablet for tablet 4bdd4b0614004b5199cd85cdd403ec75 (DEFAULT_TABLE table=test-table [id=49ced53625b944a58f20168d8680c592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:38.419034 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bdd4b0614004b5199cd85cdd403ec75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:38.420938 2618 tablet_bootstrap.cc:492] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46: Bootstrap starting.
I20260924 02:57:38.421465 2618 tablet_bootstrap.cc:654] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:38.422703 2527 tablet_service.cc:1467] Processing CreateTablet for tablet 4bdd4b0614004b5199cd85cdd403ec75 (DEFAULT_TABLE table=test-table [id=49ced53625b944a58f20168d8680c592]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:38.422817 2527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4bdd4b0614004b5199cd85cdd403ec75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:38.422811 2618 tablet_bootstrap.cc:492] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46: No bootstrap required, opened a new log
I20260924 02:57:38.422851 2618 ts_tablet_manager.cc:1397] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:38.422978 2618 raft_consensus.cc:348] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.423058 2618 raft_consensus.cc:374] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:38.423077 2618 raft_consensus.cc:729] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f06ca24b9a4a30a6fb7f9132678d46, State: Initialized, Role: FOLLOWER
I20260924 02:57:38.423121 2618 consensus_queue.cc:260] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.423197 2618 ts_tablet_manager.cc:1428] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:38.423866 2657 tablet_replica.cc:330] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46: stopping tablet replica
I20260924 02:57:38.423987 2656 tablet_replica.cc:330] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5: stopping tablet replica
I20260924 02:57:38.424067 2616 tablet_bootstrap.cc:492] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5: Bootstrap starting.
I20260924 02:57:38.424192 2657 raft_consensus.cc:2229] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:38.424103 2656 raft_consensus.cc:2229] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 02:57:38.424337 2656 raft_consensus.cc:2258] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:38.424403 2657 raft_consensus.cc:2258] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:38.424465 2616 tablet_bootstrap.cc:654] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:38.425046 2657 ts_tablet_manager.cc:1905] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:38.425158 2617 tablet_bootstrap.cc:492] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac: No bootstrap required, opened a new log
I20260924 02:57:38.425207 2617 ts_tablet_manager.cc:1397] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.001s
I20260924 02:57:38.425352 2617 raft_consensus.cc:348] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.425423 2617 raft_consensus.cc:374] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:38.425448 2617 raft_consensus.cc:729] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab79c7b14e14289a287475db0dd24ac, State: Initialized, Role: FOLLOWER
I20260924 02:57:38.425410 2656 ts_tablet_manager.cc:1905] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:38.425498 2617 consensus_queue.cc:260] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.425576 2617 ts_tablet_manager.cc:1428] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:38.427481 2616 tablet_bootstrap.cc:492] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5: No bootstrap required, opened a new log
I20260924 02:57:38.427533 2616 ts_tablet_manager.cc:1397] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260924 02:57:38.427678 2616 raft_consensus.cc:348] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.427745 2616 raft_consensus.cc:374] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:38.427769 2616 raft_consensus.cc:729] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eecc9817184a5ab3ad91ecc028abd5, State: Initialized, Role: FOLLOWER
I20260924 02:57:38.427814 2616 consensus_queue.cc:260] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.427896 2616 ts_tablet_manager.cc:1428] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:38.428617 2657 ts_tablet_manager.cc:1918] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 02:57:38.428696 2657 log.cc:1196] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/84fb04a777624facadc33a0205dc02a5
I20260924 02:57:38.428934 2657 ts_tablet_manager.cc:1939] T 84fb04a777624facadc33a0205dc02a5 P 83f06ca24b9a4a30a6fb7f9132678d46: Deleting consensus metadata
I20260924 02:57:38.429337 2098 catalog_manager.cc:4841] TS 83f06ca24b9a4a30a6fb7f9132678d46 (127.0.98.2:35033): tablet 84fb04a777624facadc33a0205dc02a5 (table test-table [id=fef31da88b734a059472923744eb989f]) successfully deleted
I20260924 02:57:38.429525 2656 ts_tablet_manager.cc:1918] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 02:57:38.429589 2656 log.cc:1196] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/84fb04a777624facadc33a0205dc02a5
I20260924 02:57:38.429845 2656 ts_tablet_manager.cc:1939] T 84fb04a777624facadc33a0205dc02a5 P 23eecc9817184a5ab3ad91ecc028abd5: Deleting consensus metadata
I20260924 02:57:38.430265 2099 catalog_manager.cc:4841] TS 23eecc9817184a5ab3ad91ecc028abd5 (127.0.98.3:46009): tablet 84fb04a777624facadc33a0205dc02a5 (table test-table [id=fef31da88b734a059472923744eb989f]) successfully deleted
I20260924 02:57:38.531541 2626 raft_consensus.cc:482] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:57:38.531628 2626 raft_consensus.cc:504] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.531780 2626 leader_election.cc:288] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dab79c7b14e14289a287475db0dd24ac (127.0.98.1:40985), 83f06ca24b9a4a30a6fb7f9132678d46 (127.0.98.2:35033)
I20260924 02:57:38.531960 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdd4b0614004b5199cd85cdd403ec75" candidate_uuid: "23eecc9817184a5ab3ad91ecc028abd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79c7b14e14289a287475db0dd24ac" is_pre_election: true
I20260924 02:57:38.532055 2270 raft_consensus.cc:2454] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eecc9817184a5ab3ad91ecc028abd5 in term 0.
I20260924 02:57:38.532039 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdd4b0614004b5199cd85cdd403ec75" candidate_uuid: "23eecc9817184a5ab3ad91ecc028abd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" is_pre_election: true
I20260924 02:57:38.532125 2409 raft_consensus.cc:2454] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eecc9817184a5ab3ad91ecc028abd5 in term 0.
I20260924 02:57:38.532210 2484 leader_election.cc:302] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eecc9817184a5ab3ad91ecc028abd5, dab79c7b14e14289a287475db0dd24ac; no voters:
I20260924 02:57:38.532298 2626 raft_consensus.cc:2781] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:57:38.532334 2626 raft_consensus.cc:482] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:57:38.532353 2626 raft_consensus.cc:3037] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:38.533120 2626 raft_consensus.cc:504] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.533247 2626 leader_election.cc:288] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [CANDIDATE]: Term 1 election: Requested vote from peers dab79c7b14e14289a287475db0dd24ac (127.0.98.1:40985), 83f06ca24b9a4a30a6fb7f9132678d46 (127.0.98.2:35033)
I20260924 02:57:38.533383 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdd4b0614004b5199cd85cdd403ec75" candidate_uuid: "23eecc9817184a5ab3ad91ecc028abd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79c7b14e14289a287475db0dd24ac"
I20260924 02:57:38.533452 2270 raft_consensus.cc:3037] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:38.533439 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4bdd4b0614004b5199cd85cdd403ec75" candidate_uuid: "23eecc9817184a5ab3ad91ecc028abd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f06ca24b9a4a30a6fb7f9132678d46"
I20260924 02:57:38.533547 2409 raft_consensus.cc:3037] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:38.533998 2270 raft_consensus.cc:2454] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eecc9817184a5ab3ad91ecc028abd5 in term 1.
I20260924 02:57:38.534147 2484 leader_election.cc:302] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eecc9817184a5ab3ad91ecc028abd5, dab79c7b14e14289a287475db0dd24ac; no voters:
I20260924 02:57:38.534224 2626 raft_consensus.cc:2781] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:38.534265 2626 raft_consensus.cc:686] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 LEADER]: Becoming Leader. State: Replica: 23eecc9817184a5ab3ad91ecc028abd5, State: Running, Role: LEADER
I20260924 02:57:38.534329 2409 raft_consensus.cc:2454] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eecc9817184a5ab3ad91ecc028abd5 in term 1.
I20260924 02:57:38.534313 2626 consensus_queue.cc:237] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:38.534885 2111 catalog_manager.cc:5495] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23eecc9817184a5ab3ad91ecc028abd5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "23eecc9817184a5ab3ad91ecc028abd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260924 02:57:38.566406 2270 raft_consensus.cc:1261] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac [term 1 FOLLOWER]: Refusing update from remote peer 23eecc9817184a5ab3ad91ecc028abd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:38.566563 2409 raft_consensus.cc:1261] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 FOLLOWER]: Refusing update from remote peer 23eecc9817184a5ab3ad91ecc028abd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:38.566634 2626 consensus_queue.cc:1035] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:38.566835 2626 consensus_queue.cc:1035] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 02:57:38.570806 2641 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 02:57:40.087273 2113 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:40.087370 2113 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38044:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:40.088119 2113 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306: Sending DeleteTablet for 3 replicas of tablet 4bdd4b0614004b5199cd85cdd403ec75
I20260924 02:57:40.088358 2390 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bdd4b0614004b5199cd85cdd403ec75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43932
I20260924 02:57:40.088376 2250 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bdd4b0614004b5199cd85cdd403ec75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42434
I20260924 02:57:40.088562 2686 tablet_replica.cc:330] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac: stopping tablet replica
I20260924 02:57:40.088624 2686 raft_consensus.cc:2229] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:40.088672 2686 raft_consensus.cc:2258] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:40.088775 2685 tablet_replica.cc:330] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46: stopping tablet replica
I20260924 02:57:40.088832 2685 raft_consensus.cc:2229] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:40.088867 2685 raft_consensus.cc:2258] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:40.088920 2686 ts_tablet_manager.cc:1905] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:40.089059 2685 ts_tablet_manager.cc:1905] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:40.090656 2527 tablet_service.cc:1514] Processing DeleteTablet for tablet 4bdd4b0614004b5199cd85cdd403ec75 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37238
I20260924 02:57:40.090823 2686 ts_tablet_manager.cc:1918] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 02:57:40.090850 2685 ts_tablet_manager.cc:1918] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 02:57:40.090878 2686 log.cc:1196] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/wal/wals/4bdd4b0614004b5199cd85cdd403ec75
I20260924 02:57:40.090906 2685 log.cc:1196] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/4bdd4b0614004b5199cd85cdd403ec75
I20260924 02:57:40.091071 2686 ts_tablet_manager.cc:1939] T 4bdd4b0614004b5199cd85cdd403ec75 P dab79c7b14e14289a287475db0dd24ac: Deleting consensus metadata
I20260924 02:57:40.091099 2685 ts_tablet_manager.cc:1939] T 4bdd4b0614004b5199cd85cdd403ec75 P 83f06ca24b9a4a30a6fb7f9132678d46: Deleting consensus metadata
I20260924 02:57:40.091377 2098 catalog_manager.cc:4841] TS 83f06ca24b9a4a30a6fb7f9132678d46 (127.0.98.2:35033): tablet 4bdd4b0614004b5199cd85cdd403ec75 (table test-table [id=49ced53625b944a58f20168d8680c592]) successfully deleted
I20260924 02:57:40.091389 2100 catalog_manager.cc:4841] TS dab79c7b14e14289a287475db0dd24ac (127.0.98.1:40985): tablet 4bdd4b0614004b5199cd85cdd403ec75 (table test-table [id=49ced53625b944a58f20168d8680c592]) successfully deleted
I20260924 02:57:40.091522 2688 tablet_replica.cc:330] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5: stopping tablet replica
I20260924 02:57:40.091574 2688 raft_consensus.cc:2229] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 02:57:40.091630 2688 raft_consensus.cc:2258] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:40.091828 2688 ts_tablet_manager.cc:1905] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:40.093391 2688 ts_tablet_manager.cc:1918] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 02:57:40.093436 2688 log.cc:1196] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/4bdd4b0614004b5199cd85cdd403ec75
I20260924 02:57:40.093604 2688 ts_tablet_manager.cc:1939] T 4bdd4b0614004b5199cd85cdd403ec75 P 23eecc9817184a5ab3ad91ecc028abd5: Deleting consensus metadata
I20260924 02:57:40.093942 2099 catalog_manager.cc:4841] TS 23eecc9817184a5ab3ad91ecc028abd5 (127.0.98.3:46009): tablet 4bdd4b0614004b5199cd85cdd403ec75 (table test-table [id=49ced53625b944a58f20168d8680c592]) successfully deleted
Sep 24 02:57:40 dist-test-slave-sb6p krb5kdc[2068](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218660, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 02:57:40 dist-test-slave-sb6p krb5kdc[2068](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218660, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:40.113850 2695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:40.103604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38054 (local address 127.0.98.62:46545)
0924 02:57:40.103741 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:40.103742 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:40.103751 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:57:40.103803 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:40.103803 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:40.103814 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:40.103871 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:40.104403 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.104901 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.105598 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.105784 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.105826 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:40.111868 (+ 6042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:40.111869 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:40.111869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:40.111876 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:40.113017 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:40.113295 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:40.113296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:40.113296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:40.113341 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:40.113460 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:40.113461 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:40.113461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:40.113555 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:40.113622 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:40.113715 (+ 93us) server_negotiation.cc:299] Negotiation successful
0924 02:57:40.113777 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
I20260924 02:57:40.115207 2113 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 02:57:40.115347 2113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:57:40.116523 2250 tablet_service.cc:1467] Processing CreateTablet for tablet 47c9feb169b14462b384d78748cf6343 (DEFAULT_TABLE table=test-table [id=b8db4520cd614b25b50957d220a48615]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:40.116520 2527 tablet_service.cc:1467] Processing CreateTablet for tablet 47c9feb169b14462b384d78748cf6343 (DEFAULT_TABLE table=test-table [id=b8db4520cd614b25b50957d220a48615]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:40.116530 2390 tablet_service.cc:1467] Processing CreateTablet for tablet 47c9feb169b14462b384d78748cf6343 (DEFAULT_TABLE table=test-table [id=b8db4520cd614b25b50957d220a48615]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:40.116643 2250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c9feb169b14462b384d78748cf6343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:40.116650 2527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c9feb169b14462b384d78748cf6343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:40.116672 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 47c9feb169b14462b384d78748cf6343. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:40.117637 2696 tablet_bootstrap.cc:492] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac: Bootstrap starting.
I20260924 02:57:40.117925 2697 tablet_bootstrap.cc:492] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46: Bootstrap starting.
I20260924 02:57:40.117971 2696 tablet_bootstrap.cc:654] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:40.118319 2697 tablet_bootstrap.cc:654] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:40.118598 2696 tablet_bootstrap.cc:492] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac: No bootstrap required, opened a new log
I20260924 02:57:40.118629 2698 tablet_bootstrap.cc:492] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5: Bootstrap starting.
I20260924 02:57:40.118654 2696 ts_tablet_manager.cc:1397] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:40.118796 2696 raft_consensus.cc:348] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.118870 2696 raft_consensus.cc:374] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:40.118896 2696 raft_consensus.cc:729] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab79c7b14e14289a287475db0dd24ac, State: Initialized, Role: FOLLOWER
I20260924 02:57:40.118934 2696 consensus_queue.cc:260] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.119022 2698 tablet_bootstrap.cc:654] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:40.119207 2696 ts_tablet_manager.cc:1428] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:40.119258 2697 tablet_bootstrap.cc:492] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46: No bootstrap required, opened a new log
I20260924 02:57:40.119297 2697 ts_tablet_manager.cc:1397] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:40.119433 2697 raft_consensus.cc:348] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.119513 2697 raft_consensus.cc:374] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:40.119509 2698 tablet_bootstrap.cc:492] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5: No bootstrap required, opened a new log
I20260924 02:57:40.119532 2697 raft_consensus.cc:729] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f06ca24b9a4a30a6fb7f9132678d46, State: Initialized, Role: FOLLOWER
I20260924 02:57:40.119553 2698 ts_tablet_manager.cc:1397] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:40.119572 2697 consensus_queue.cc:260] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.119729 2697 ts_tablet_manager.cc:1428] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:40.119750 2698 raft_consensus.cc:348] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.119817 2698 raft_consensus.cc:374] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:40.119834 2698 raft_consensus.cc:729] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eecc9817184a5ab3ad91ecc028abd5, State: Initialized, Role: FOLLOWER
I20260924 02:57:40.119966 2698 consensus_queue.cc:260] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.120070 2698 ts_tablet_manager.cc:1428] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:40.144331 2668 raft_consensus.cc:482] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:57:40.144413 2668 raft_consensus.cc:504] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.144560 2668 leader_election.cc:288] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dab79c7b14e14289a287475db0dd24ac (127.0.98.1:40985), 83f06ca24b9a4a30a6fb7f9132678d46 (127.0.98.2:35033)
I20260924 02:57:40.144732 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c9feb169b14462b384d78748cf6343" candidate_uuid: "23eecc9817184a5ab3ad91ecc028abd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79c7b14e14289a287475db0dd24ac" is_pre_election: true
I20260924 02:57:40.144779 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c9feb169b14462b384d78748cf6343" candidate_uuid: "23eecc9817184a5ab3ad91ecc028abd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" is_pre_election: true
I20260924 02:57:40.144820 2270 raft_consensus.cc:2454] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eecc9817184a5ab3ad91ecc028abd5 in term 0.
I20260924 02:57:40.144850 2409 raft_consensus.cc:2454] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 23eecc9817184a5ab3ad91ecc028abd5 in term 0.
I20260924 02:57:40.144976 2484 leader_election.cc:302] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eecc9817184a5ab3ad91ecc028abd5, dab79c7b14e14289a287475db0dd24ac; no voters:
I20260924 02:57:40.145097 2668 raft_consensus.cc:2781] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:57:40.145126 2668 raft_consensus.cc:482] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:57:40.145139 2668 raft_consensus.cc:3037] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:40.145742 2668 raft_consensus.cc:504] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.145841 2668 leader_election.cc:288] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [CANDIDATE]: Term 1 election: Requested vote from peers dab79c7b14e14289a287475db0dd24ac (127.0.98.1:40985), 83f06ca24b9a4a30a6fb7f9132678d46 (127.0.98.2:35033)
I20260924 02:57:40.146011 2270 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c9feb169b14462b384d78748cf6343" candidate_uuid: "23eecc9817184a5ab3ad91ecc028abd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79c7b14e14289a287475db0dd24ac"
I20260924 02:57:40.146085 2270 raft_consensus.cc:3037] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:40.145989 2409 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "47c9feb169b14462b384d78748cf6343" candidate_uuid: "23eecc9817184a5ab3ad91ecc028abd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83f06ca24b9a4a30a6fb7f9132678d46"
I20260924 02:57:40.146118 2409 raft_consensus.cc:3037] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:40.146828 2270 raft_consensus.cc:2454] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eecc9817184a5ab3ad91ecc028abd5 in term 1.
I20260924 02:57:40.146857 2409 raft_consensus.cc:2454] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 23eecc9817184a5ab3ad91ecc028abd5 in term 1.
I20260924 02:57:40.146983 2482 leader_election.cc:302] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eecc9817184a5ab3ad91ecc028abd5, 83f06ca24b9a4a30a6fb7f9132678d46; no voters:
I20260924 02:57:40.147076 2668 raft_consensus.cc:2781] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:40.147125 2668 raft_consensus.cc:686] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 LEADER]: Becoming Leader. State: Replica: 23eecc9817184a5ab3ad91ecc028abd5, State: Running, Role: LEADER
I20260924 02:57:40.147182 2668 consensus_queue.cc:237] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.147718 2113 catalog_manager.cc:5495] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 23eecc9817184a5ab3ad91ecc028abd5 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "23eecc9817184a5ab3ad91ecc028abd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } health_report { overall_health: HEALTHY } } }
I20260924 02:57:40.156633 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:40.154535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37260 (local address 127.0.98.3:46009)
0924 02:57:40.154649 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:40.154650 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:40.154659 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:57:40.154679 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:40.154679 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:40.154688 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:40.154742 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:40.154831 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.155365 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.156106 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.156299 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.156365 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:40.156457 (+ 92us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:40.156484 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:40.156520 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:40.156558 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 02:57:40.156576 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":71,"threads_started":1}
I20260924 02:57:40.158411 2409 raft_consensus.cc:1261] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 FOLLOWER]: Refusing update from remote peer 23eecc9817184a5ab3ad91ecc028abd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:40.158427 2270 raft_consensus.cc:1261] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac [term 1 FOLLOWER]: Refusing update from remote peer 23eecc9817184a5ab3ad91ecc028abd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:40.158633 2680 consensus_queue.cc:1035] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:40.158764 2668 consensus_queue.cc:1035] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:40.167413 2111 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:40.167500 2111 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:40.168001 2111 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306: Sending DeleteTablet for 3 replicas of tablet 47c9feb169b14462b384d78748cf6343
I20260924 02:57:40.168277 2390 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c9feb169b14462b384d78748cf6343 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43932
I20260924 02:57:40.168293 2527 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c9feb169b14462b384d78748cf6343 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37238
I20260924 02:57:40.168367 2685 tablet_replica.cc:330] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46: stopping tablet replica
I20260924 02:57:40.168381 2688 tablet_replica.cc:330] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5: stopping tablet replica
I20260924 02:57:40.168370 2250 tablet_service.cc:1514] Processing DeleteTablet for tablet 47c9feb169b14462b384d78748cf6343 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:40 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42434
I20260924 02:57:40.168412 2685 raft_consensus.cc:2229] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:40.168426 2688 raft_consensus.cc:2229] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 LEADER]: Raft consensus shutting down.
I20260924 02:57:40.168439 2686 tablet_replica.cc:330] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac: stopping tablet replica
I20260924 02:57:40.168447 2685 raft_consensus.cc:2258] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:40.168470 2688 raft_consensus.cc:2258] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:40.168478 2686 raft_consensus.cc:2229] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:40.168509 2686 raft_consensus.cc:2258] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:40.168721 2111 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38054:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 02:57:40.168794 2685 ts_tablet_manager.cc:1905] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:40.168809 2686 ts_tablet_manager.cc:1905] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 02:57:40.168841 2111 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:57:40.169759 2250 tablet_service.cc:1467] Processing CreateTablet for tablet e2ed2ba8cc33455c8a7efb033b806aa4 (DEFAULT_TABLE table=test-table [id=0be3e761351e4351b0bba45500428cee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:40.169773 2390 tablet_service.cc:1467] Processing CreateTablet for tablet e2ed2ba8cc33455c8a7efb033b806aa4 (DEFAULT_TABLE table=test-table [id=0be3e761351e4351b0bba45500428cee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:40.169858 2250 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ed2ba8cc33455c8a7efb033b806aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:40.169858 2390 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ed2ba8cc33455c8a7efb033b806aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:40.170231 2688 ts_tablet_manager.cc:1905] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:40.170385 2686 ts_tablet_manager.cc:1918] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 02:57:40.170429 2686 log.cc:1196] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-0/wal/wals/47c9feb169b14462b384d78748cf6343
I20260924 02:57:40.170634 2686 ts_tablet_manager.cc:1939] T 47c9feb169b14462b384d78748cf6343 P dab79c7b14e14289a287475db0dd24ac: Deleting consensus metadata
I20260924 02:57:40.170648 2685 ts_tablet_manager.cc:1918] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 02:57:40.170713 2685 log.cc:1196] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/47c9feb169b14462b384d78748cf6343
I20260924 02:57:40.170904 2685 ts_tablet_manager.cc:1939] T 47c9feb169b14462b384d78748cf6343 P 83f06ca24b9a4a30a6fb7f9132678d46: Deleting consensus metadata
I20260924 02:57:40.171015 2527 tablet_service.cc:1467] Processing CreateTablet for tablet e2ed2ba8cc33455c8a7efb033b806aa4 (DEFAULT_TABLE table=test-table [id=0be3e761351e4351b0bba45500428cee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:40.171116 2527 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e2ed2ba8cc33455c8a7efb033b806aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:40.171154 2098 catalog_manager.cc:4841] TS 83f06ca24b9a4a30a6fb7f9132678d46 (127.0.98.2:35033): tablet 47c9feb169b14462b384d78748cf6343 (table test-table [id=b8db4520cd614b25b50957d220a48615]) successfully deleted
I20260924 02:57:40.171636 2688 ts_tablet_manager.cc:1918] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 02:57:40.171689 2688 log.cc:1196] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/47c9feb169b14462b384d78748cf6343
I20260924 02:57:40.171695 2697 tablet_bootstrap.cc:492] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46: Bootstrap starting.
I20260924 02:57:40.171787 2696 tablet_bootstrap.cc:492] T e2ed2ba8cc33455c8a7efb033b806aa4 P dab79c7b14e14289a287475db0dd24ac: Bootstrap starting.
I20260924 02:57:40.171854 2688 ts_tablet_manager.cc:1939] T 47c9feb169b14462b384d78748cf6343 P 23eecc9817184a5ab3ad91ecc028abd5: Deleting consensus metadata
I20260924 02:57:40.172082 2697 tablet_bootstrap.cc:654] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:40.172175 2698 tablet_bootstrap.cc:492] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5: Bootstrap starting.
I20260924 02:57:40.172194 2696 tablet_bootstrap.cc:654] T e2ed2ba8cc33455c8a7efb033b806aa4 P dab79c7b14e14289a287475db0dd24ac: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:40.172441 2698 tablet_bootstrap.cc:654] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:40.172483 2099 catalog_manager.cc:4841] TS 23eecc9817184a5ab3ad91ecc028abd5 (127.0.98.3:46009): tablet 47c9feb169b14462b384d78748cf6343 (table test-table [id=b8db4520cd614b25b50957d220a48615]) successfully deleted
I20260924 02:57:40.172549 2100 catalog_manager.cc:4841] TS dab79c7b14e14289a287475db0dd24ac (127.0.98.1:40985): tablet 47c9feb169b14462b384d78748cf6343 (table test-table [id=b8db4520cd614b25b50957d220a48615]) successfully deleted
I20260924 02:57:40.172942 2698 tablet_bootstrap.cc:492] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5: No bootstrap required, opened a new log
I20260924 02:57:40.172943 2696 tablet_bootstrap.cc:492] T e2ed2ba8cc33455c8a7efb033b806aa4 P dab79c7b14e14289a287475db0dd24ac: No bootstrap required, opened a new log
I20260924 02:57:40.172964 2697 tablet_bootstrap.cc:492] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46: No bootstrap required, opened a new log
I20260924 02:57:40.172986 2698 ts_tablet_manager.cc:1397] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:40.172986 2696 ts_tablet_manager.cc:1397] T e2ed2ba8cc33455c8a7efb033b806aa4 P dab79c7b14e14289a287475db0dd24ac: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:40.173005 2697 ts_tablet_manager.cc:1397] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:40.173085 2698 raft_consensus.cc:348] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.173105 2696 raft_consensus.cc:348] T e2ed2ba8cc33455c8a7efb033b806aa4 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.173151 2696 raft_consensus.cc:374] T e2ed2ba8cc33455c8a7efb033b806aa4 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:40.173142 2698 raft_consensus.cc:374] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:40.173170 2696 raft_consensus.cc:729] T e2ed2ba8cc33455c8a7efb033b806aa4 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab79c7b14e14289a287475db0dd24ac, State: Initialized, Role: FOLLOWER
I20260924 02:57:40.173173 2698 raft_consensus.cc:729] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23eecc9817184a5ab3ad91ecc028abd5, State: Initialized, Role: FOLLOWER
I20260924 02:57:40.173141 2697 raft_consensus.cc:348] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.173197 2697 raft_consensus.cc:374] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:40.173214 2697 raft_consensus.cc:729] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83f06ca24b9a4a30a6fb7f9132678d46, State: Initialized, Role: FOLLOWER
I20260924 02:57:40.173214 2696 consensus_queue.cc:260] T e2ed2ba8cc33455c8a7efb033b806aa4 P dab79c7b14e14289a287475db0dd24ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.173214 2698 consensus_queue.cc:260] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.173252 2697 consensus_queue.cc:260] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.173293 2696 ts_tablet_manager.cc:1428] T e2ed2ba8cc33455c8a7efb033b806aa4 P dab79c7b14e14289a287475db0dd24ac: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:40.173293 2698 ts_tablet_manager.cc:1428] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:40.173331 2697 ts_tablet_manager.cc:1428] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:40.282861 2703 raft_consensus.cc:482] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:57:40.282949 2703 raft_consensus.cc:504] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.283232 2703 leader_election.cc:288] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dab79c7b14e14289a287475db0dd24ac (127.0.98.1:40985), 23eecc9817184a5ab3ad91ecc028abd5 (127.0.98.3:46009)
I20260924 02:57:40.286270 2715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:40.283459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60323 (local address 127.0.98.1:40985)
0924 02:57:40.283589 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:40.283591 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:40.283601 (+ 10us) server_negotiation.cc:407] Connection header received
0924 02:57:40.283692 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:40.283693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:40.283704 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:40.283763 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:40.283926 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.284455 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.285436 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.285969 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.286029 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:40.286039 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:40.286183 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 02:57:40.286209 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":49,"threads_started":1}
I20260924 02:57:40.286320 2714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:40.283401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40985 (local address 127.0.98.2:60323)
0924 02:57:40.283549 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:40.283560 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:40.283614 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:40.283768 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:40.283769 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:40.283771 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:40.283846 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:40.283848 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.284562 (+ 714us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.284563 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:40.285350 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:40.285352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.286043 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.286044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:40.286118 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:40.286123 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 02:57:40.286198 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 02:57:40.286236 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":91,"threads_started":1}
I20260924 02:57:40.286586 2269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ed2ba8cc33455c8a7efb033b806aa4" candidate_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79c7b14e14289a287475db0dd24ac" is_pre_election: true
I20260924 02:57:40.286665 2713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:40.283404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46009 (local address 127.0.98.2:43019)
0924 02:57:40.283560 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:40.283570 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:40.283648 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:40.283838 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:40.283839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:40.283840 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:40.283891 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:40.283892 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.284942 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.284943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:40.285777 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:40.285779 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.286430 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.286430 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:40.286488 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:40.286491 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:40.286563 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 02:57:40.286594 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":73,"threads_started":1}
I20260924 02:57:40.286684 2269 raft_consensus.cc:2454] T e2ed2ba8cc33455c8a7efb033b806aa4 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f06ca24b9a4a30a6fb7f9132678d46 in term 0.
I20260924 02:57:40.286689 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:40.283464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43019 (local address 127.0.98.3:46009)
0924 02:57:40.283515 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:40.283517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:40.283622 (+ 105us) server_negotiation.cc:407] Connection header received
0924 02:57:40.283751 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:40.283752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:40.283761 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:40.283801 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:40.283975 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.284449 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.285862 (+ 1413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.286343 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.286403 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:40.286414 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:40.286573 (+ 159us) server_negotiation.cc:299] Negotiation successful
0924 02:57:40.286612 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 02:57:40.286882 2344 leader_election.cc:302] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 83f06ca24b9a4a30a6fb7f9132678d46, dab79c7b14e14289a287475db0dd24ac; no voters:
I20260924 02:57:40.286902 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ed2ba8cc33455c8a7efb033b806aa4" candidate_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eecc9817184a5ab3ad91ecc028abd5" is_pre_election: true
I20260924 02:57:40.286994 2703 raft_consensus.cc:2781] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:57:40.287026 2703 raft_consensus.cc:482] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:57:40.287030 2550 raft_consensus.cc:2454] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 83f06ca24b9a4a30a6fb7f9132678d46 in term 0.
I20260924 02:57:40.287050 2703 raft_consensus.cc:3037] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:40.287714 2703 raft_consensus.cc:504] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.287823 2703 leader_election.cc:288] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [CANDIDATE]: Term 1 election: Requested vote from peers dab79c7b14e14289a287475db0dd24ac (127.0.98.1:40985), 23eecc9817184a5ab3ad91ecc028abd5 (127.0.98.3:46009)
I20260924 02:57:40.287994 2550 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ed2ba8cc33455c8a7efb033b806aa4" candidate_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23eecc9817184a5ab3ad91ecc028abd5"
I20260924 02:57:40.288002 2269 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e2ed2ba8cc33455c8a7efb033b806aa4" candidate_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79c7b14e14289a287475db0dd24ac"
I20260924 02:57:40.288051 2550 raft_consensus.cc:3037] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:40.288064 2269 raft_consensus.cc:3037] T e2ed2ba8cc33455c8a7efb033b806aa4 P dab79c7b14e14289a287475db0dd24ac [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:40.288611 2550 raft_consensus.cc:2454] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f06ca24b9a4a30a6fb7f9132678d46 in term 1.
I20260924 02:57:40.288721 2269 raft_consensus.cc:2454] T e2ed2ba8cc33455c8a7efb033b806aa4 P dab79c7b14e14289a287475db0dd24ac [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 83f06ca24b9a4a30a6fb7f9132678d46 in term 1.
I20260924 02:57:40.288785 2343 leader_election.cc:302] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23eecc9817184a5ab3ad91ecc028abd5, 83f06ca24b9a4a30a6fb7f9132678d46; no voters:
I20260924 02:57:40.288865 2703 raft_consensus.cc:2781] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:40.288983 2703 raft_consensus.cc:686] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 LEADER]: Becoming Leader. State: Replica: 83f06ca24b9a4a30a6fb7f9132678d46, State: Running, Role: LEADER
I20260924 02:57:40.289048 2703 consensus_queue.cc:237] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } }
I20260924 02:57:40.289654 2113 catalog_manager.cc:5495] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 reported cstate change: term changed from 0 to 1, leader changed from <none> to 83f06ca24b9a4a30a6fb7f9132678d46 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "83f06ca24b9a4a30a6fb7f9132678d46" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 35033 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 } health_report { overall_health: UNKNOWN } } }
I20260924 02:57:40.318389 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:40.316009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43946 (local address 127.0.98.2:35033)
0924 02:57:40.316186 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:40.316188 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:40.316194 (+ 6us) server_negotiation.cc:407] Connection header received
0924 02:57:40.316229 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:40.316230 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:40.316248 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:40.316314 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:40.316405 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.316879 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.317544 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.317751 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.317800 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:40.317896 (+ 96us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:40.317997 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:40.318043 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:40.318073 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 02:57:40.318088 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":117,"threads_started":1}
I20260924 02:57:40.322584 2706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:40.320646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38627 (local address 127.0.98.3:46009)
0924 02:57:40.320712 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:40.320714 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:40.320785 (+ 71us) server_negotiation.cc:407] Connection header received
0924 02:57:40.320830 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:40.320831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:40.320838 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:40.320881 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:40.320978 (+ 97us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.321410 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.322056 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.322234 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.322277 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:40.322404 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:40.322428 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:40.322460 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:40.322499 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 02:57:40.322530 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260924 02:57:40.322667 2721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:40.320583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46009 (local address 127.0.98.2:38627)
0924 02:57:40.320737 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:40.320746 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:40.320794 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:40.320888 (+ 94us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:40.320889 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:40.320890 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 02:57:40.320941 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:40.320942 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.321455 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.321455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:40.321999 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:40.322001 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.322298 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.322299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:40.322339 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:40.322350 (+ 11us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 02:57:40.322467 (+ 117us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 02:57:40.322468 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 02:57:40.322497 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 02:57:40.322557 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":72,"threads_started":1}
I20260924 02:57:40.324391 2550 raft_consensus.cc:1261] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 FOLLOWER]: Refusing update from remote peer 83f06ca24b9a4a30a6fb7f9132678d46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:40.324390 2269 raft_consensus.cc:1261] T e2ed2ba8cc33455c8a7efb033b806aa4 P dab79c7b14e14289a287475db0dd24ac [term 1 FOLLOWER]: Refusing update from remote peer 83f06ca24b9a4a30a6fb7f9132678d46: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:40.324613 2716 consensus_queue.cc:1035] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab79c7b14e14289a287475db0dd24ac" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 40985 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:40.324738 2703 consensus_queue.cc:1035] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23eecc9817184a5ab3ad91ecc028abd5" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46009 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:40.331200 2725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:40.328949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:35033 (local address 127.0.98.3:50517)
0924 02:57:40.329083 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:40.329095 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:40.329141 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:40.329254 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:40.329255 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:40.329256 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 02:57:40.329302 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:40.329304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.329840 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.329840 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:40.330458 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:40.330459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.330906 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.330907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:40.330962 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:40.330967 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 02:57:40.331084 (+ 117us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 02:57:40.331085 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 02:57:40.331112 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 02:57:40.331143 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":79,"threads_started":1}
I20260924 02:57:40.331202 2718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:40.329127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50517 (local address 127.0.98.2:35033)
0924 02:57:40.329165 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:40.329167 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:40.329174 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:57:40.329196 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:40.329197 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:40.329207 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:40.329253 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:40.329344 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.329790 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.330572 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:40.330844 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:40.330906 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:40.331027 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:40.331051 (+ 24us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:40.331083 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:40.331114 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 02:57:40.331129 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 02:57:40.331529 2669 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260924 02:57:41.848541 2112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:41.848640 2112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:38054:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:41.849310 2112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 3286545749e34697a142d5d482720306: Sending DeleteTablet for 3 replicas of tablet e2ed2ba8cc33455c8a7efb033b806aa4
I20260924 02:57:41.849522 2250 tablet_service.cc:1514] Processing DeleteTablet for tablet e2ed2ba8cc33455c8a7efb033b806aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:42434
I20260924 02:57:41.849558 2527 tablet_service.cc:1514] Processing DeleteTablet for tablet e2ed2ba8cc33455c8a7efb033b806aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37238
I20260924 02:57:41.849548 2390 tablet_service.cc:1514] Processing DeleteTablet for tablet e2ed2ba8cc33455c8a7efb033b806aa4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:41 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43932
I20260924 02:57:41.849686 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 2185
I20260924 02:57:41.849738 2752 tablet_replica.cc:330] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46: stopping tablet replica
I20260924 02:57:41.850260 2752 raft_consensus.cc:2229] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 LEADER]: Raft consensus shutting down.
I20260924 02:57:41.850344 2752 raft_consensus.cc:2258] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:41.850579 2752 ts_tablet_manager.cc:1905] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:41.850636 2751 tablet_replica.cc:330] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5: stopping tablet replica
I20260924 02:57:41.850692 2751 raft_consensus.cc:2229] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:41.850724 2751 raft_consensus.cc:2258] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:41.850958 2751 ts_tablet_manager.cc:1905] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:41.852422 2751 ts_tablet_manager.cc:1918] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 02:57:41.852421 2752 ts_tablet_manager.cc:1918] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260924 02:57:41.852463 2751 log.cc:1196] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/e2ed2ba8cc33455c8a7efb033b806aa4
I20260924 02:57:41.852468 2752 log.cc:1196] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/e2ed2ba8cc33455c8a7efb033b806aa4
I20260924 02:57:41.852605 2751 ts_tablet_manager.cc:1939] T e2ed2ba8cc33455c8a7efb033b806aa4 P 23eecc9817184a5ab3ad91ecc028abd5: Deleting consensus metadata
I20260924 02:57:41.852712 2752 ts_tablet_manager.cc:1939] T e2ed2ba8cc33455c8a7efb033b806aa4 P 83f06ca24b9a4a30a6fb7f9132678d46: Deleting consensus metadata
I20260924 02:57:41.852862 2099 catalog_manager.cc:4841] TS 23eecc9817184a5ab3ad91ecc028abd5 (127.0.98.3:46009): tablet e2ed2ba8cc33455c8a7efb033b806aa4 (table test-table [id=0be3e761351e4351b0bba45500428cee]) successfully deleted
I20260924 02:57:41.852984 2098 catalog_manager.cc:4841] TS 83f06ca24b9a4a30a6fb7f9132678d46 (127.0.98.2:35033): tablet e2ed2ba8cc33455c8a7efb033b806aa4 (table test-table [id=0be3e761351e4351b0bba45500428cee]) successfully deleted
W20260924 02:57:41.854384 2100 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:40985 (error 108)
W20260924 02:57:41.855319 2754 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:57:41.854749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:40985 (local address 127.0.0.1:42446)
0924 02:57:41.854874 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:41.854887 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:41.854935 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:41.855272 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40985: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
W20260924 02:57:41.855427 2100 catalog_manager.cc:4584] TS dab79c7b14e14289a287475db0dd24ac (127.0.98.1:40985): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e2ed2ba8cc33455c8a7efb033b806aa4: Network error: Client connection negotiation failed: client connection to 127.0.98.1:40985: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 02:57:41.855479 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 2325
I20260924 02:57:41.861267 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 2465
I20260924 02:57:41.865775 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 2081
2026-09-24T02:57:41Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5523 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:57:41 dist-test-slave-sb6p krb5kdc[2757](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:57:41 dist-test-slave-sb6p krb5kdc[2757](info): set up 2 sockets
Sep 24 02:57:41 dist-test-slave-sb6p krb5kdc[2757](info): commencing operation
krb5kdc: starting...
W20260924 02:57:42.818120 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:57:42 dist-test-slave-sb6p krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:57:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:57:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 02:57:42.927388 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42743
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:42743
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:43.005172 2770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:43.005366 2770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:43.005388 2770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:43.006485 2770 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:43.006543 2770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:43.006556 2770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:43.006574 2770 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:43.006585 2770 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:57:43.008028 2770 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:42743
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:42743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2770
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:43.008212 2770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:43.008378 2770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:43.012996 2778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:43.012965 2776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:43.012985 2775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:43.013249 2770 server_base.cc:1034] running on GCE node
I20260924 02:57:43.013476 2770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:43.013727 2770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:43.014853 2770 hybrid_clock.cc:648] HybridClock initialized: now 1790218663014845 us; error 41 us; skew 500 ppm
Sep 24 02:57:43 dist-test-slave-sb6p krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:43.017359 2770 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:57:43.017789 2770 webserver.cc:460] Webserver started at http://127.0.98.62:42339/ using document root <none> and password file <none>
I20260924 02:57:43.018002 2770 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:43.018057 2770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:43.018179 2770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:43.019241 2770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "8d0a1f98c535429ebdf837daa275ca9c"
format_stamp: "Formatted at 2026-09-24 02:57:43 on dist-test-slave-sb6p"
server_key: "09e473696bc3edb69b48fad28e743b34"
server_key_iv: "8ff92562e01903de17f724cfb3bfd3b2"
server_key_version: "encryptionkey@0"
I20260924 02:57:43.019559 2770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "8d0a1f98c535429ebdf837daa275ca9c"
format_stamp: "Formatted at 2026-09-24 02:57:43 on dist-test-slave-sb6p"
server_key: "09e473696bc3edb69b48fad28e743b34"
server_key_iv: "8ff92562e01903de17f724cfb3bfd3b2"
server_key_version: "encryptionkey@0"
I20260924 02:57:43.020714 2770 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 02:57:43.021410 2785 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:43.021567 2770 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:43.021641 2770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "8d0a1f98c535429ebdf837daa275ca9c"
format_stamp: "Formatted at 2026-09-24 02:57:43 on dist-test-slave-sb6p"
server_key: "09e473696bc3edb69b48fad28e743b34"
server_key_iv: "8ff92562e01903de17f724cfb3bfd3b2"
server_key_version: "encryptionkey@0"
I20260924 02:57:43.021704 2770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:43.041440 2770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:43.044111 2770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:43.044266 2770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:43.047938 2770 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:42743
I20260924 02:57:43.047972 2837 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:42743 every 8 connection(s)
I20260924 02:57:43.048439 2770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:57:43.049088 2838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:43.051347 2838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c: Bootstrap starting.
I20260924 02:57:43.051597 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 2770
I20260924 02:57:43.051693 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:57:43.051859 392 external_mini_cluster.cc:1428] Setting key 23ce594341e9c79cb162d0f8a45e111e
I20260924 02:57:43.052016 2838 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:43.052276 2838 log.cc:824] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:43.052960 2838 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c: No bootstrap required, opened a new log
I20260924 02:57:43.054802 2838 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0a1f98c535429ebdf837daa275ca9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } }
I20260924 02:57:43.054936 2838 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:43.054967 2838 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d0a1f98c535429ebdf837daa275ca9c, State: Initialized, Role: FOLLOWER
I20260924 02:57:43.055094 2838 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0a1f98c535429ebdf837daa275ca9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } }
I20260924 02:57:43.055171 2838 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:57:43.055200 2838 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:57:43.055245 2838 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:43.055734 2838 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0a1f98c535429ebdf837daa275ca9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } }
I20260924 02:57:43.055825 2838 leader_election.cc:302] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d0a1f98c535429ebdf837daa275ca9c; no voters:
I20260924 02:57:43.055967 2838 leader_election.cc:288] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:57:43.056008 2843 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:43.056211 2843 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [term 1 LEADER]: Becoming Leader. State: Replica: 8d0a1f98c535429ebdf837daa275ca9c, State: Running, Role: LEADER
I20260924 02:57:43.056216 2838 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:43.056357 2843 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0a1f98c535429ebdf837daa275ca9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } }
I20260924 02:57:43.056751 2844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d0a1f98c535429ebdf837daa275ca9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0a1f98c535429ebdf837daa275ca9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } }
I20260924 02:57:43.056831 2844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:43.057063 2845 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d0a1f98c535429ebdf837daa275ca9c. Latest consensus state: current_term: 1 leader_uuid: "8d0a1f98c535429ebdf837daa275ca9c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d0a1f98c535429ebdf837daa275ca9c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 42743 } } }
I20260924 02:57:43.057364 2845 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:43.057552 2852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:43.058050 2852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 02:57:43 dist-test-slave-sb6p krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218662, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:43.059840 2852 catalog_manager.cc:1347] Generated new cluster ID: 6e5345ef24a349f891b143f648d4bb64
I20260924 02:57:43.059885 2852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:57:43.062688 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:43.052486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58052 (local address 127.0.98.62:42743)
0924 02:57:43.052636 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:43.052644 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:43.052667 (+ 23us) server_negotiation.cc:407] Connection header received
0924 02:57:43.053093 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:43.053101 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:43.053163 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:43.053286 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:43.053866 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:43.054770 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:43.055744 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:43.056318 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:43.057994 (+ 1676us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:43.059090 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:43.059094 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:43.059094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:43.059105 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:43.060923 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:43.061300 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:43.061303 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:43.061303 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:43.061391 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:43.061550 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:43.061550 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:43.061550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:43.061816 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:43.061945 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:43.062330 (+ 385us) server_negotiation.cc:299] Negotiation successful
0924 02:57:43.062462 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":78,"threads_started":1}
I20260924 02:57:43.069188 2852 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:57:43.069700 2852 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:43.072773 2852 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d0a1f98c535429ebdf837daa275ca9c: Generated new TSK 0
I20260924 02:57:43.072939 2852 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 02:57:43.096720 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42743
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:43.173882 2864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:43.174067 2864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:43.174095 2864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:43.175259 2864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:43.175298 2864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:43.175333 2864 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:57:43.176770 2864 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:43.176982 2864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:43.177173 2864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:43.181847 2870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:43.181856 2872 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:43.181856 2869 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:43.181943 2864 server_base.cc:1034] running on GCE node
I20260924 02:57:43.182266 2864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:43.182454 2864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:43.183579 2864 hybrid_clock.cc:648] HybridClock initialized: now 1790218663183566 us; error 29 us; skew 500 ppm
Sep 24 02:57:43 dist-test-slave-sb6p krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:43.185798 2864 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 02:57:43.186229 2864 webserver.cc:460] Webserver started at http://127.0.98.1:34029/ using document root <none> and password file <none>
I20260924 02:57:43.186393 2864 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:43.186430 2864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:43.186537 2864 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:43.187520 2864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "cfe89cd4634b46a9b8082cfc8d36c0d3"
format_stamp: "Formatted at 2026-09-24 02:57:43 on dist-test-slave-sb6p"
server_key: "b82bfeaca3fc5b2cadf67a30b309e8e4"
server_key_iv: "099b4cbebfe7989e2b4941b0e589b743"
server_key_version: "encryptionkey@0"
I20260924 02:57:43.187798 2864 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "cfe89cd4634b46a9b8082cfc8d36c0d3"
format_stamp: "Formatted at 2026-09-24 02:57:43 on dist-test-slave-sb6p"
server_key: "b82bfeaca3fc5b2cadf67a30b309e8e4"
server_key_iv: "099b4cbebfe7989e2b4941b0e589b743"
server_key_version: "encryptionkey@0"
I20260924 02:57:43.188890 2864 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:57:43.189499 2879 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:43.189634 2864 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:43.189713 2864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "cfe89cd4634b46a9b8082cfc8d36c0d3"
format_stamp: "Formatted at 2026-09-24 02:57:43 on dist-test-slave-sb6p"
server_key: "b82bfeaca3fc5b2cadf67a30b309e8e4"
server_key_iv: "099b4cbebfe7989e2b4941b0e589b743"
server_key_version: "encryptionkey@0"
I20260924 02:57:43.189776 2864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:43.202488 2864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:43.204110 2864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:43.204221 2864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:43.204393 2864 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:43.204629 2864 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:43.204658 2864 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:43.204677 2864 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:43.204689 2864 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:43.209940 2864 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36503
I20260924 02:57:43.209990 2992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36503 every 8 connection(s)
I20260924 02:57:43.210270 2864 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:57:43.211349 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 2864
I20260924 02:57:43.211438 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:57:43.211603 392 external_mini_cluster.cc:1428] Setting key 9201d48689d6710687dc501a9923c2ce
Sep 24 02:57:43 dist-test-slave-sb6p krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 02:57:43.219934 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:43.210920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60935 (local address 127.0.98.62:42743)
0924 02:57:43.210980 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:43.210982 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:43.211169 (+ 187us) server_negotiation.cc:407] Connection header received
0924 02:57:43.212435 (+ 1266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:43.212436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:43.212445 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:43.212480 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:43.213254 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:43.213933 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:43.214994 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:43.215296 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:43.215370 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:43.217368 (+ 1998us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:43.217370 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:43.217370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:43.217378 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:43.218798 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:43.219141 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:43.219142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:43.219143 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:43.219208 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:43.219405 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:43.219406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:43.219406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:43.219536 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:43.219601 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:43.219807 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 02:57:43.219871 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 02:57:43.220089 2995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:43.210898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42743 (local address 127.0.98.1:60935)
0924 02:57:43.211045 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:43.211085 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:43.212344 (+ 1259us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:43.212564 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:43.212568 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:43.212902 (+ 334us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:43.213164 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:43.213170 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:43.214019 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:43.214020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:43.214916 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:43.214918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:43.215368 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:43.215369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:43.215493 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:43.215965 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:43.215973 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:43.217281 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:43.218870 (+ 1589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:43.218876 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:43.218877 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:43.219075 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:43.219261 (+ 186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:43.219262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:43.219262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:43.219356 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:43.219590 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:43.219593 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:43.219708 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 02:57:43.219784 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 02:57:43.219896 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":79,"threads_started":1}
I20260924 02:57:43.220546 2993 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42743
I20260924 02:57:43.220631 2993 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:43.220795 2993 heartbeater.cc:511] Master 127.0.98.62:42743 requested a full tablet report, sending...
I20260924 02:57:43.221304 2802 ts_manager.cc:194] Registered new tserver with Master: cfe89cd4634b46a9b8082cfc8d36c0d3 (127.0.98.1:36503)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 02:57:43.221983 2802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 02:57:43.234028 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42743
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:43.310814 2998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:43.311019 2998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:43.311048 2998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:43.312179 2998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:43.312219 2998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:43.312254 2998 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:57:43.313730 2998 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:43.313962 2998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:43.314162 2998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:43.318905 3004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:43.318899 3006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:43.318910 3003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:43.318936 2998 server_base.cc:1034] running on GCE node
I20260924 02:57:43.319288 2998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:43.319506 2998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:43.320643 2998 hybrid_clock.cc:648] HybridClock initialized: now 1790218663320628 us; error 26 us; skew 500 ppm
Sep 24 02:57:43 dist-test-slave-sb6p krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:43.322997 2998 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 02:57:43.323412 2998 webserver.cc:460] Webserver started at http://127.0.98.2:44709/ using document root <none> and password file <none>
I20260924 02:57:43.323606 2998 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:43.323652 2998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:43.323779 2998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:43.324800 2998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "d41358d407d24ca2a724d80617211e10"
format_stamp: "Formatted at 2026-09-24 02:57:43 on dist-test-slave-sb6p"
server_key: "eb1b58e926350b31831c021e30e726bd"
server_key_iv: "ed0f7c694de9d925c3235612fc067076"
server_key_version: "encryptionkey@0"
I20260924 02:57:43.325111 2998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d41358d407d24ca2a724d80617211e10"
format_stamp: "Formatted at 2026-09-24 02:57:43 on dist-test-slave-sb6p"
server_key: "eb1b58e926350b31831c021e30e726bd"
server_key_iv: "ed0f7c694de9d925c3235612fc067076"
server_key_version: "encryptionkey@0"
I20260924 02:57:43.326258 2998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:57:43.326964 3013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:43.327109 2998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:43.327193 2998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "d41358d407d24ca2a724d80617211e10"
format_stamp: "Formatted at 2026-09-24 02:57:43 on dist-test-slave-sb6p"
server_key: "eb1b58e926350b31831c021e30e726bd"
server_key_iv: "ed0f7c694de9d925c3235612fc067076"
server_key_version: "encryptionkey@0"
I20260924 02:57:43.327257 2998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:43.334039 2998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:43.335768 2998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:43.335891 2998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:43.336069 2998 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:43.336321 2998 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:43.336352 2998 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:43.336378 2998 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:43.336401 2998 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:43.341585 2998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42763
I20260924 02:57:43.341620 3126 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42763 every 8 connection(s)
I20260924 02:57:43.341912 2998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 02:57:43 dist-test-slave-sb6p krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:43.347733 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 2998
I20260924 02:57:43.347829 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:57:43.347992 392 external_mini_cluster.cc:1428] Setting key c13172c30c1f211ba93628341acd0c97
I20260924 02:57:43.350608 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:43.342586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35667 (local address 127.0.98.62:42743)
0924 02:57:43.342649 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:43.342651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:43.342810 (+ 159us) server_negotiation.cc:407] Connection header received
0924 02:57:43.343703 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:43.343705 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:43.343715 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:43.343765 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:43.344309 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:43.344799 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:43.345721 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:43.345898 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:43.345941 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:43.347912 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:43.347914 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:43.347914 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:43.347922 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:43.349171 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:43.349618 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:43.349618 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:43.349619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:43.349666 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:43.349916 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:43.349917 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:43.349917 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:43.350054 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:43.350118 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:43.350426 (+ 308us) server_negotiation.cc:299] Negotiation successful
0924 02:57:43.350501 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 02:57:43.350786 3129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:43.342555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42743 (local address 127.0.98.2:35667)
0924 02:57:43.342696 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:43.342734 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:43.343602 (+ 868us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:43.343819 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:43.343823 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:43.344062 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:43.344251 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:43.344254 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:43.344858 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:43.344858 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:43.345658 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:43.345661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:43.345952 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:43.345952 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:43.346051 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:43.346463 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:43.346471 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:43.347793 (+ 1322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:43.349247 (+ 1454us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:43.349254 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:43.349256 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:43.349530 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:43.349717 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:43.349718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:43.349718 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:43.349847 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:43.350166 (+ 319us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:43.350172 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:43.350300 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 02:57:43.350401 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 02:57:43.350570 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":70,"threads_started":1}
I20260924 02:57:43.351207 3127 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42743
I20260924 02:57:43.351293 3127 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:43.351450 3127 heartbeater.cc:511] Master 127.0.98.62:42743 requested a full tablet report, sending...
I20260924 02:57:43.351779 2802 ts_manager.cc:194] Registered new tserver with Master: d41358d407d24ca2a724d80617211e10 (127.0.98.2:42763)
I20260924 02:57:43.352303 2802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35667
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 02:57:43.371245 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42743
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39387
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:43.449674 3132 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:43.449862 3132 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:43.449882 3132 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:43.451054 3132 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:43.451103 3132 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:43.451139 3132 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:57:43.452836 3132 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39387
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:42743
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3132
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:43.453023 3132 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:43.453203 3132 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:43.458081 3140 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:43.458215 3132 server_base.cc:1034] running on GCE node
W20260924 02:57:43.458094 3137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:43.458101 3138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:43.458509 3132 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:43.458772 3132 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:43.459930 3132 hybrid_clock.cc:648] HybridClock initialized: now 1790218663459909 us; error 33 us; skew 500 ppm
Sep 24 02:57:43 dist-test-slave-sb6p krb5kdc[2757](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:43.462384 3132 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 02:57:43.462857 3132 webserver.cc:460] Webserver started at http://127.0.98.3:44725/ using document root <none> and password file <none>
I20260924 02:57:43.463068 3132 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:43.463117 3132 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:43.463236 3132 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:43.464226 3132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "609f69dd6779470199d8ea86c3f61e72"
format_stamp: "Formatted at 2026-09-24 02:57:43 on dist-test-slave-sb6p"
server_key: "d556a979c88d0c9a8bcfa3ae331311dd"
server_key_iv: "c1505b5bba4d82dc628530bbb3bc1c1c"
server_key_version: "encryptionkey@0"
I20260924 02:57:43.464533 3132 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "609f69dd6779470199d8ea86c3f61e72"
format_stamp: "Formatted at 2026-09-24 02:57:43 on dist-test-slave-sb6p"
server_key: "d556a979c88d0c9a8bcfa3ae331311dd"
server_key_iv: "c1505b5bba4d82dc628530bbb3bc1c1c"
server_key_version: "encryptionkey@0"
I20260924 02:57:43.465745 3132 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 02:57:43.466409 3147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:43.466581 3132 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:43.466662 3132 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "609f69dd6779470199d8ea86c3f61e72"
format_stamp: "Formatted at 2026-09-24 02:57:43 on dist-test-slave-sb6p"
server_key: "d556a979c88d0c9a8bcfa3ae331311dd"
server_key_iv: "c1505b5bba4d82dc628530bbb3bc1c1c"
server_key_version: "encryptionkey@0"
I20260924 02:57:43.466725 3132 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:43.487506 3132 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:43.489234 3132 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:43.489341 3132 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:43.489508 3132 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:43.489774 3132 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:43.489801 3132 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:43.489827 3132 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:43.489841 3132 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:43.495213 3132 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38429
I20260924 02:57:43.495287 3260 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38429 every 8 connection(s)
I20260924 02:57:43.495558 3132 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 02:57:43 dist-test-slave-sb6p krb5kdc[2757](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:43.503943 2842 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:43.496250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34151 (local address 127.0.98.62:42743)
0924 02:57:43.496315 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:43.496317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:43.496490 (+ 173us) server_negotiation.cc:407] Connection header received
0924 02:57:43.497405 (+ 915us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:43.497406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:43.497418 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:43.497469 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:43.498067 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:43.498614 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:43.499570 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:43.499777 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:43.499830 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:43.501805 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:43.501806 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:43.501806 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:43.501813 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:43.502917 (+ 1104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:43.503254 (+ 337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:43.503255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:43.503255 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:43.503301 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:43.503469 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:43.503470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:43.503470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:43.503561 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:43.503628 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:43.503821 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 02:57:43.503878 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 02:57:43.504096 3263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:43.496212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:42743 (local address 127.0.98.3:34151)
0924 02:57:43.496375 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:43.496412 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:43.497306 (+ 894us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:43.497541 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:43.497545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:43.497800 (+ 255us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:43.497985 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:43.497989 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:43.498688 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:43.498689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:43.499493 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:43.499495 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:43.499843 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:43.499844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:43.499961 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:43.500428 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:43.500436 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:43.501711 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:43.502989 (+ 1278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:43.502994 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:43.502996 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:43.503193 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:43.503346 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:43.503346 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:43.503347 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:43.503430 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:43.503622 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:43.503624 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:43.503731 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 02:57:43.503803 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 02:57:43.503913 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":84,"threads_started":1}
I20260924 02:57:43.504492 3261 heartbeater.cc:347] Connected to a master server at 127.0.98.62:42743
I20260924 02:57:43.504578 3261 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:43.504720 3261 heartbeater.cc:511] Master 127.0.98.62:42743 requested a full tablet report, sending...
I20260924 02:57:43.505081 2802 ts_manager.cc:194] Registered new tserver with Master: 609f69dd6779470199d8ea86c3f61e72 (127.0.98.3:38429)
I20260924 02:57:43.505434 2802 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:34151
I20260924 02:57:43.505474 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 3132
I20260924 02:57:43.505560 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:57:43.505756 392 external_mini_cluster.cc:1428] Setting key ff7c8353e2a726b0a1e5898419393bf7
I20260924 02:57:43.506552 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 02:57:43.516512 2842 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:57:43.515693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58066 (local address 127.0.98.62:42743)
0924 02:57:43.515742 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:43.515743 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:43.515833 (+ 90us) server_negotiation.cc:407] Connection header received
0924 02:57:43.515878 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:43.515879 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:43.515888 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:43.515940 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:43.516430 (+ 490us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58066: BlockingRecv error: recv got EOF from 127.0.0.1:58066 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 02:57:43.517501 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 2864
I20260924 02:57:43.521850 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 2998
I20260924 02:57:43.525367 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 3132
I20260924 02:57:43.529297 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 2770
2026-09-24T02:57:43Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1662 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:57:43 dist-test-slave-sb6p krb5kdc[3273](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:57:43 dist-test-slave-sb6p krb5kdc[3273](info): set up 2 sockets
Sep 24 02:57:43 dist-test-slave-sb6p krb5kdc[3273](info): commencing operation
krb5kdc: starting...
W20260924 02:57:44.482048 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:57:44 dist-test-slave-sb6p krb5kdc[3273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:57:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:57:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 02:57:44.589165 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35145
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35145
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:44.666329 3286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:44.666572 3286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:44.666606 3286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:44.667765 3286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:44.667804 3286 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:44.667821 3286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:44.667838 3286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:44.667852 3286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:57:44.669327 3286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35145
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3286
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:44.669567 3286 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:44.669768 3286 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:44.674268 3294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:44.674295 3291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:44.674350 3292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:44.674392 3286 server_base.cc:1034] running on GCE node
I20260924 02:57:44.674739 3286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:44.674979 3286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:44.676131 3286 hybrid_clock.cc:648] HybridClock initialized: now 1790218664676115 us; error 29 us; skew 500 ppm
Sep 24 02:57:44 dist-test-slave-sb6p krb5kdc[3273](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218664, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:44.678402 3286 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:57:44.678821 3286 webserver.cc:460] Webserver started at http://127.0.98.62:46851/ using document root <none> and password file <none>
I20260924 02:57:44.679006 3286 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:44.679045 3286 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:44.679131 3286 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:44.680140 3286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "24e561361e5a432eae8ee69e42a21a11"
format_stamp: "Formatted at 2026-09-24 02:57:44 on dist-test-slave-sb6p"
server_key: "f25aee447f4c6beaac84065a879af2c8"
server_key_iv: "7c891852ff242bb4521bc00e02bfb8a6"
server_key_version: "encryptionkey@0"
I20260924 02:57:44.680426 3286 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "24e561361e5a432eae8ee69e42a21a11"
format_stamp: "Formatted at 2026-09-24 02:57:44 on dist-test-slave-sb6p"
server_key: "f25aee447f4c6beaac84065a879af2c8"
server_key_iv: "7c891852ff242bb4521bc00e02bfb8a6"
server_key_version: "encryptionkey@0"
I20260924 02:57:44.681545 3286 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 02:57:44.682133 3301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:44.682292 3286 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:44.682374 3286 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "24e561361e5a432eae8ee69e42a21a11"
format_stamp: "Formatted at 2026-09-24 02:57:44 on dist-test-slave-sb6p"
server_key: "f25aee447f4c6beaac84065a879af2c8"
server_key_iv: "7c891852ff242bb4521bc00e02bfb8a6"
server_key_version: "encryptionkey@0"
I20260924 02:57:44.682431 3286 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:44.692914 3286 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:44.694619 3286 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:44.694747 3286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:44.698204 3286 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35145
I20260924 02:57:44.698261 3353 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35145 every 8 connection(s)
I20260924 02:57:44.698591 3286 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:57:44.699445 3354 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:44.701576 3354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11: Bootstrap starting.
I20260924 02:57:44.702127 3354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:44.702392 3354 log.cc:824] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:44.702982 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 3286
I20260924 02:57:44.703080 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:57:44.703081 3354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11: No bootstrap required, opened a new log
I20260924 02:57:44.703254 392 external_mini_cluster.cc:1428] Setting key d870c46e556641c086ae2c70adb0d8e2
I20260924 02:57:44.705706 3354 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e561361e5a432eae8ee69e42a21a11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } }
I20260924 02:57:44.705835 3354 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:44.705856 3354 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e561361e5a432eae8ee69e42a21a11, State: Initialized, Role: FOLLOWER
I20260924 02:57:44.705967 3354 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e561361e5a432eae8ee69e42a21a11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } }
I20260924 02:57:44.706035 3354 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:57:44.706074 3354 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:57:44.706120 3354 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:44.706763 3354 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e561361e5a432eae8ee69e42a21a11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } }
I20260924 02:57:44.706867 3354 leader_election.cc:302] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24e561361e5a432eae8ee69e42a21a11; no voters:
I20260924 02:57:44.707013 3354 leader_election.cc:288] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:57:44.707053 3359 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:44.707190 3359 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [term 1 LEADER]: Becoming Leader. State: Replica: 24e561361e5a432eae8ee69e42a21a11, State: Running, Role: LEADER
I20260924 02:57:44.707244 3354 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:44.707314 3359 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e561361e5a432eae8ee69e42a21a11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } }
I20260924 02:57:44.707674 3359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24e561361e5a432eae8ee69e42a21a11. Latest consensus state: current_term: 1 leader_uuid: "24e561361e5a432eae8ee69e42a21a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e561361e5a432eae8ee69e42a21a11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } }
I20260924 02:57:44.707721 3361 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24e561361e5a432eae8ee69e42a21a11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e561361e5a432eae8ee69e42a21a11" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35145 } } }
I20260924 02:57:44.707830 3359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:44.707836 3361 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:44.708086 3368 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:44.708776 3368 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 02:57:44 dist-test-slave-sb6p krb5kdc[3273](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218664, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:44.710414 3368 catalog_manager.cc:1347] Generated new cluster ID: a49221b37ad14a7ab3de0a7bfff6a983
I20260924 02:57:44.710466 3368 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:57:44.713498 3358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:44.703694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39224 (local address 127.0.98.62:35145)
0924 02:57:44.703873 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:44.703881 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:44.703904 (+ 23us) server_negotiation.cc:407] Connection header received
0924 02:57:44.704394 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:44.704400 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:44.704447 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:44.704554 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:44.705283 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:44.706279 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:44.707327 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:44.707990 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:44.708078 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:44.710741 (+ 2663us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:44.710744 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:44.710745 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:44.710756 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:44.712191 (+ 1435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:44.712505 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:44.712508 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:44.712508 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:44.712583 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:44.712750 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:44.712751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:44.712751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:44.712965 (+ 214us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:44.713099 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:44.713219 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 02:57:44.713345 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":91,"threads_started":1}
I20260924 02:57:44.714453 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 02:57:44.716187 3368 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:57:44.716820 3368 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:44.723054 3368 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24e561361e5a432eae8ee69e42a21a11: Generated new TSK 0
I20260924 02:57:44.723210 3368 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 02:57:44.729487 3358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:44.725891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39232 (local address 127.0.98.62:35145)
0924 02:57:44.725942 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:44.725944 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:44.726010 (+ 66us) server_negotiation.cc:407] Connection header received
0924 02:57:44.726101 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:44.726101 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:44.726109 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:44.726161 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:44.726610 (+ 449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:44.727122 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:44.728198 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:44.728402 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:44.728453 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:44.729092 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:44.729093 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:44.729093 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:57:44.729274 (+ 181us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:44.729289 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:57:44.729294 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:44.729339 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:44.729408 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 02:57:44.729431 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 02:57:44.731887 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 3286
2026-09-24T02:57:44Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1201 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:57:44 dist-test-slave-sb6p krb5kdc[3387](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:57:44 dist-test-slave-sb6p krb5kdc[3387](info): set up 2 sockets
Sep 24 02:57:44 dist-test-slave-sb6p krb5kdc[3387](info): commencing operation
krb5kdc: starting...
W20260924 02:57:45.670135 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:57:45 dist-test-slave-sb6p krb5kdc[3387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:57:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:57:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 02:57:45.780776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34181
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34181
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:45.860646 3400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:45.860853 3400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:45.860874 3400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:45.862016 3400 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:45.862061 3400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:45.862074 3400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:45.862094 3400 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:45.862105 3400 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:57:45.863595 3400 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34181
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3400
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:45.863822 3400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:45.864034 3400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:45.868705 3406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:45.868713 3405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:45.868778 3400 server_base.cc:1034] running on GCE node
W20260924 02:57:45.868713 3408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:45.869166 3400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:45.869427 3400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:45.870561 3400 hybrid_clock.cc:648] HybridClock initialized: now 1790218665870551 us; error 29 us; skew 500 ppm
Sep 24 02:57:45 dist-test-slave-sb6p krb5kdc[3387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:45.876101 3400 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:57:45.876529 3400 webserver.cc:460] Webserver started at http://127.0.98.62:43359/ using document root <none> and password file <none>
I20260924 02:57:45.876730 3400 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:45.876778 3400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:45.876900 3400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:45.877965 3400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "ea2dc255074f4c48a3948108fd00043e"
format_stamp: "Formatted at 2026-09-24 02:57:45 on dist-test-slave-sb6p"
server_key: "95489e8618b9e7c2fd90fa4c4af9371f"
server_key_iv: "d40a444352773b289d4f5e8e28bf5be8"
server_key_version: "encryptionkey@0"
I20260924 02:57:45.878284 3400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "ea2dc255074f4c48a3948108fd00043e"
format_stamp: "Formatted at 2026-09-24 02:57:45 on dist-test-slave-sb6p"
server_key: "95489e8618b9e7c2fd90fa4c4af9371f"
server_key_iv: "d40a444352773b289d4f5e8e28bf5be8"
server_key_version: "encryptionkey@0"
I20260924 02:57:45.879550 3400 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 02:57:45.880316 3415 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:45.880475 3400 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:45.880558 3400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "ea2dc255074f4c48a3948108fd00043e"
format_stamp: "Formatted at 2026-09-24 02:57:45 on dist-test-slave-sb6p"
server_key: "95489e8618b9e7c2fd90fa4c4af9371f"
server_key_iv: "d40a444352773b289d4f5e8e28bf5be8"
server_key_version: "encryptionkey@0"
I20260924 02:57:45.880621 3400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:45.896422 3400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:45.898190 3400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:45.898321 3400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:45.901791 3400 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34181
I20260924 02:57:45.901849 3467 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34181 every 8 connection(s)
I20260924 02:57:45.902149 3400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:57:45.903102 3468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:45.905114 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 3400
I20260924 02:57:45.905205 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:57:45.905323 3468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e: Bootstrap starting.
I20260924 02:57:45.905349 392 external_mini_cluster.cc:1428] Setting key bf62b4ac3293cde8d7bad06660d31d35
I20260924 02:57:45.905890 3468 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:45.906178 3468 log.cc:824] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:45.906792 3468 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e: No bootstrap required, opened a new log
I20260924 02:57:45.908468 3468 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2dc255074f4c48a3948108fd00043e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } }
I20260924 02:57:45.908598 3468 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:45.908629 3468 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea2dc255074f4c48a3948108fd00043e, State: Initialized, Role: FOLLOWER
I20260924 02:57:45.908749 3468 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2dc255074f4c48a3948108fd00043e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } }
I20260924 02:57:45.908828 3468 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:57:45.908859 3468 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:57:45.908903 3468 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:45.909530 3468 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2dc255074f4c48a3948108fd00043e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } }
I20260924 02:57:45.909638 3468 leader_election.cc:302] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea2dc255074f4c48a3948108fd00043e; no voters:
I20260924 02:57:45.909806 3468 leader_election.cc:288] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:57:45.909822 3473 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:45.910050 3468 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:45.910256 3473 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [term 1 LEADER]: Becoming Leader. State: Replica: ea2dc255074f4c48a3948108fd00043e, State: Running, Role: LEADER
I20260924 02:57:45.910368 3473 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2dc255074f4c48a3948108fd00043e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } }
I20260924 02:57:45.910714 3474 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea2dc255074f4c48a3948108fd00043e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2dc255074f4c48a3948108fd00043e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } } }
I20260924 02:57:45.910712 3473 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea2dc255074f4c48a3948108fd00043e. Latest consensus state: current_term: 1 leader_uuid: "ea2dc255074f4c48a3948108fd00043e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea2dc255074f4c48a3948108fd00043e" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34181 } } }
I20260924 02:57:45.910892 3474 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:45.910957 3473 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:45.911237 3482 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:45.911705 3482 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 02:57:45 dist-test-slave-sb6p krb5kdc[3387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:45.913142 3482 catalog_manager.cc:1347] Generated new cluster ID: 7236001095af428680809019271d9bb6
I20260924 02:57:45.913211 3482 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:57:45.915508 3471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:45.905822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36686 (local address 127.0.98.62:34181)
0924 02:57:45.905976 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:45.905986 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:45.906016 (+ 30us) server_negotiation.cc:407] Connection header received
0924 02:57:45.906512 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:45.906539 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:45.906599 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:45.906723 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:45.907320 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:45.908153 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:45.908930 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:45.909534 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:45.909663 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:45.912282 (+ 2619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:45.912286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:45.912286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:45.912298 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:45.914155 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:45.914470 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:45.914473 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:45.914473 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:45.914592 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:45.914769 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:45.914770 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:45.914770 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:45.914992 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:45.915133 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:45.915227 (+ 94us) server_negotiation.cc:299] Negotiation successful
0924 02:57:45.915354 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260924 02:57:45.944586 3482 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:57:45.945138 3482 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:45.949455 3482 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea2dc255074f4c48a3948108fd00043e: Generated new TSK 0
I20260924 02:57:45.949636 3482 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 02:57:45.989372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34181
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:46.075838 3494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:46.076050 3494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:46.076073 3494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:46.077245 3494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:46.077281 3494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:46.077307 3494 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:57:46.078701 3494 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3494
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:46.078894 3494 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:46.079077 3494 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 02:57:46.083875 3494 server_base.cc:1034] running on GCE node
W20260924 02:57:46.083874 3502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:46.083992 3500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:46.084059 3499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:46.084280 3494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:46.084506 3494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:46.085635 3494 hybrid_clock.cc:648] HybridClock initialized: now 1790218666085627 us; error 28 us; skew 500 ppm
Sep 24 02:57:46 dist-test-slave-sb6p krb5kdc[3387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:46.087958 3494 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 02:57:46.088369 3494 webserver.cc:460] Webserver started at http://127.0.98.1:38965/ using document root <none> and password file <none>
I20260924 02:57:46.088555 3494 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:46.088601 3494 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:46.088718 3494 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:46.089722 3494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "5828f05a6e164c0ea856a24ef340e5be"
format_stamp: "Formatted at 2026-09-24 02:57:46 on dist-test-slave-sb6p"
server_key: "d80d532134392b5df50ec44dd85a14aa"
server_key_iv: "8d4c8ff60735edcbad6e19b273764595"
server_key_version: "encryptionkey@0"
I20260924 02:57:46.090051 3494 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "5828f05a6e164c0ea856a24ef340e5be"
format_stamp: "Formatted at 2026-09-24 02:57:46 on dist-test-slave-sb6p"
server_key: "d80d532134392b5df50ec44dd85a14aa"
server_key_iv: "8d4c8ff60735edcbad6e19b273764595"
server_key_version: "encryptionkey@0"
I20260924 02:57:46.091297 3494 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 02:57:46.091955 3509 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:46.092168 3494 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:46.092247 3494 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "5828f05a6e164c0ea856a24ef340e5be"
format_stamp: "Formatted at 2026-09-24 02:57:46 on dist-test-slave-sb6p"
server_key: "d80d532134392b5df50ec44dd85a14aa"
server_key_iv: "8d4c8ff60735edcbad6e19b273764595"
server_key_version: "encryptionkey@0"
I20260924 02:57:46.092308 3494 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:46.107326 3494 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:46.109197 3494 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:46.109332 3494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:46.109565 3494 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:46.109861 3494 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:46.109894 3494 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:46.109922 3494 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:46.109943 3494 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:46.115626 3494 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43257
I20260924 02:57:46.115664 3622 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43257 every 8 connection(s)
I20260924 02:57:46.115979 3494 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 02:57:46 dist-test-slave-sb6p krb5kdc[3387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:46.124349 3471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:46.116654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38487 (local address 127.0.98.62:34181)
0924 02:57:46.116730 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:46.116733 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:46.116887 (+ 154us) server_negotiation.cc:407] Connection header received
0924 02:57:46.117393 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:46.117394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:46.117404 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:46.117465 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:46.118345 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:46.118911 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:46.119882 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:46.120081 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:46.120132 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:46.122055 (+ 1923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:46.122056 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:46.122056 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:46.122062 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:46.123299 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:46.123639 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:46.123640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:46.123640 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:46.123691 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:46.123874 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:46.123875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:46.123875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:46.123967 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:46.124010 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:46.124212 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 02:57:46.124282 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 02:57:46.124632 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 3494
I20260924 02:57:46.124709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:57:46.124861 392 external_mini_cluster.cc:1428] Setting key f227790b1e130177df24ee67f2703e80
I20260924 02:57:46.125003 3625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:46.116629 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.1:38487)
0924 02:57:46.116768 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:46.116806 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:46.117289 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:46.117523 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:46.117526 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:46.118064 (+ 538us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:46.118253 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:46.118268 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:46.118986 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:46.118987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:46.119807 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:46.119809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:46.120143 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:46.120143 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:46.120239 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:46.120709 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:46.120717 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:46.121981 (+ 1264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:46.123363 (+ 1382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:46.123370 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:46.123371 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:46.123579 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:46.123739 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:46.123740 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:46.123740 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:46.123832 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:46.124007 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:46.124010 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:46.124122 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 02:57:46.124197 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 02:57:46.124318 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260924 02:57:46.125450 3623 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34181
I20260924 02:57:46.125555 3623 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:46.125741 3623 heartbeater.cc:511] Master 127.0.98.62:34181 requested a full tablet report, sending...
I20260924 02:57:46.126279 3432 ts_manager.cc:194] Registered new tserver with Master: 5828f05a6e164c0ea856a24ef340e5be (127.0.98.1:43257)
I20260924 02:57:46.126924 3432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:38487
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 02:57:46.148662 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34181
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:46.227298 3628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:46.227500 3628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:46.227522 3628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:46.228672 3628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:46.228708 3628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:46.228734 3628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:57:46.230087 3628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3628
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:46.230263 3628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:46.230434 3628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 02:57:46.235266 3628 server_base.cc:1034] running on GCE node
W20260924 02:57:46.235248 3634 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:46.235316 3636 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:46.235219 3633 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:46.235581 3628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:46.235777 3628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:46.236901 3628 hybrid_clock.cc:648] HybridClock initialized: now 1790218666236889 us; error 28 us; skew 500 ppm
Sep 24 02:57:46 dist-test-slave-sb6p krb5kdc[3387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:46.239223 3628 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 02:57:46.239651 3628 webserver.cc:460] Webserver started at http://127.0.98.2:32799/ using document root <none> and password file <none>
I20260924 02:57:46.239843 3628 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:46.239886 3628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:46.240001 3628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:46.240998 3628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "49bb0abfd37e4266b9098382ee10462c"
format_stamp: "Formatted at 2026-09-24 02:57:46 on dist-test-slave-sb6p"
server_key: "47c1b6da0b46e9a6ff2803aa58c0b597"
server_key_iv: "4f598dbb68c011494f3b428d1acdbd0d"
server_key_version: "encryptionkey@0"
I20260924 02:57:46.241302 3628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "49bb0abfd37e4266b9098382ee10462c"
format_stamp: "Formatted at 2026-09-24 02:57:46 on dist-test-slave-sb6p"
server_key: "47c1b6da0b46e9a6ff2803aa58c0b597"
server_key_iv: "4f598dbb68c011494f3b428d1acdbd0d"
server_key_version: "encryptionkey@0"
I20260924 02:57:46.242466 3628 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 02:57:46.243160 3643 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:46.243299 3628 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:46.243371 3628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "49bb0abfd37e4266b9098382ee10462c"
format_stamp: "Formatted at 2026-09-24 02:57:46 on dist-test-slave-sb6p"
server_key: "47c1b6da0b46e9a6ff2803aa58c0b597"
server_key_iv: "4f598dbb68c011494f3b428d1acdbd0d"
server_key_version: "encryptionkey@0"
I20260924 02:57:46.243433 3628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:46.266706 3628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:46.268428 3628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:46.268538 3628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:46.268759 3628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:46.269057 3628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:46.269089 3628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:46.269119 3628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:46.269142 3628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:46.274631 3628 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45373
I20260924 02:57:46.274684 3756 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45373 every 8 connection(s)
I20260924 02:57:46.274993 3628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 02:57:46 dist-test-slave-sb6p krb5kdc[3387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:46.283421 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 3628
I20260924 02:57:46.283533 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:57:46.283772 392 external_mini_cluster.cc:1428] Setting key 6deb9cf0216cc38cd502298072ea9fbd
I20260924 02:57:46.283830 3471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:46.275671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37561 (local address 127.0.98.62:34181)
0924 02:57:46.275730 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:46.275732 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:46.275944 (+ 212us) server_negotiation.cc:407] Connection header received
0924 02:57:46.276427 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:46.276428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:46.276438 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:46.276492 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:46.277403 (+ 911us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:46.277906 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:46.278962 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:46.279141 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:46.279190 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:46.281215 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:46.281216 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:46.281216 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:46.281223 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:46.282480 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:46.282923 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:46.282924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:46.282924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:46.282974 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:46.283161 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:46.283161 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:46.283161 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:46.283258 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:46.283303 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:46.283666 (+ 363us) server_negotiation.cc:299] Negotiation successful
0924 02:57:46.283734 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 02:57:46.284080 3759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:46.275654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.2:37561)
0924 02:57:46.275821 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:46.275864 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:46.276343 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:46.276555 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:46.276559 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:46.277122 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:46.277322 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:46.277328 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:46.278033 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:46.278034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:46.278896 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:46.278899 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:46.279222 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:46.279222 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:46.279340 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:46.279776 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:46.279784 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:46.281133 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:46.282578 (+ 1445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:46.282585 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:46.282587 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:46.282860 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:46.283024 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:46.283024 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:46.283025 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:46.283120 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:46.283312 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:46.283317 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:46.283483 (+ 166us) client_negotiation.cc:769] Sending connection context
0924 02:57:46.283651 (+ 168us) client_negotiation.cc:240] Negotiation successful
0924 02:57:46.283853 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":75,"threads_started":1}
I20260924 02:57:46.284638 3757 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34181
I20260924 02:57:46.284731 3757 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:46.284899 3757 heartbeater.cc:511] Master 127.0.98.62:34181 requested a full tablet report, sending...
I20260924 02:57:46.285231 3432 ts_manager.cc:194] Registered new tserver with Master: 49bb0abfd37e4266b9098382ee10462c (127.0.98.2:45373)
I20260924 02:57:46.285555 3432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:37561
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 02:57:46.306664 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34181
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:46.388935 3762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:46.389142 3762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:46.389165 3762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:46.390322 3762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:46.390367 3762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:46.390391 3762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:57:46.391844 3762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34181
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:46.392071 3762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:46.392289 3762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:46.397017 3768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:46.397029 3767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:46.397037 3770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:46.397239 3762 server_base.cc:1034] running on GCE node
I20260924 02:57:46.397472 3762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:46.397665 3762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:46.398801 3762 hybrid_clock.cc:648] HybridClock initialized: now 1790218666398789 us; error 25 us; skew 500 ppm
Sep 24 02:57:46 dist-test-slave-sb6p krb5kdc[3387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:46.401178 3762 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 02:57:46.401556 3762 webserver.cc:460] Webserver started at http://127.0.98.3:36629/ using document root <none> and password file <none>
I20260924 02:57:46.401777 3762 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:46.401821 3762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:46.401945 3762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:46.403064 3762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "fe0bad9c0b1e4d2593cb025515a68351"
format_stamp: "Formatted at 2026-09-24 02:57:46 on dist-test-slave-sb6p"
server_key: "a485a710157e8a1ba60170d7cb7f3b55"
server_key_iv: "70a7c2b4f8e7457ce9c27cd8cda56e72"
server_key_version: "encryptionkey@0"
I20260924 02:57:46.403386 3762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "fe0bad9c0b1e4d2593cb025515a68351"
format_stamp: "Formatted at 2026-09-24 02:57:46 on dist-test-slave-sb6p"
server_key: "a485a710157e8a1ba60170d7cb7f3b55"
server_key_iv: "70a7c2b4f8e7457ce9c27cd8cda56e72"
server_key_version: "encryptionkey@0"
I20260924 02:57:46.404559 3762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:57:46.405241 3777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:46.405392 3762 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:46.405467 3762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "fe0bad9c0b1e4d2593cb025515a68351"
format_stamp: "Formatted at 2026-09-24 02:57:46 on dist-test-slave-sb6p"
server_key: "a485a710157e8a1ba60170d7cb7f3b55"
server_key_iv: "70a7c2b4f8e7457ce9c27cd8cda56e72"
server_key_version: "encryptionkey@0"
I20260924 02:57:46.405531 3762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:46.413053 3762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:46.415849 3762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:46.415972 3762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:46.416195 3762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:46.416512 3762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:46.416546 3762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:46.416574 3762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:46.416606 3762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:46.421865 3762 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44323
I20260924 02:57:46.421922 3890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44323 every 8 connection(s)
I20260924 02:57:46.422196 3762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 02:57:46 dist-test-slave-sb6p krb5kdc[3387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:46.430666 3471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:46.422894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35009 (local address 127.0.98.62:34181)
0924 02:57:46.422968 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:46.422970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:46.423140 (+ 170us) server_negotiation.cc:407] Connection header received
0924 02:57:46.423627 (+ 487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:46.423627 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:46.423637 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:46.423690 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:46.424584 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:46.425129 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:46.426140 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:46.426328 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:46.426381 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:46.428399 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:46.428400 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:46.428400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:46.428407 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:46.429531 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:46.429882 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:46.429882 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:46.429882 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:46.429926 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:46.430106 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:46.430106 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:46.430107 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:46.430231 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:46.430279 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:46.430506 (+ 227us) server_negotiation.cc:299] Negotiation successful
0924 02:57:46.430602 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 02:57:46.430826 3893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:46.422881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34181 (local address 127.0.98.3:35009)
0924 02:57:46.423022 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:46.423057 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:46.423514 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:46.423750 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:46.423753 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:46.424303 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:46.424496 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:46.424501 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:46.425207 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:46.425208 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:46.426013 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:46.426015 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:46.426386 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:46.426387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:46.426494 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:46.427026 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:46.427034 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:46.428324 (+ 1290us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:46.429600 (+ 1276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:46.429606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:46.429607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:46.429821 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:46.429972 (+ 151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:46.429972 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:46.429972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:46.430064 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:46.430278 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:46.430282 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:46.430394 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 02:57:46.430479 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 02:57:46.430638 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260924 02:57:46.431070 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 3762
I20260924 02:57:46.431196 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:57:46.431243 3891 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34181
I20260924 02:57:46.431337 3891 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:46.431385 392 external_mini_cluster.cc:1428] Setting key 8eaf8d3a3f54a0318c2b5afde155117f
I20260924 02:57:46.431509 3891 heartbeater.cc:511] Master 127.0.98.62:34181 requested a full tablet report, sending...
I20260924 02:57:46.431864 3432 ts_manager.cc:194] Registered new tserver with Master: fe0bad9c0b1e4d2593cb025515a68351 (127.0.98.3:44323)
I20260924 02:57:46.432192 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 02:57:46.432317 3432 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:35009
Sep 24 02:57:46 dist-test-slave-sb6p krb5kdc[3387](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218666, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 24 02:57:46 dist-test-slave-sb6p krb5kdc[3387](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218666, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:46.453014 3471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:46.446662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36698 (local address 127.0.98.62:34181)
0924 02:57:46.446709 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:46.446711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:46.446851 (+ 140us) server_negotiation.cc:407] Connection header received
0924 02:57:46.446908 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:46.446909 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:46.446919 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:46.446980 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:46.447498 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:46.447983 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:46.448669 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:46.448853 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:46.448899 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:46.451056 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:46.451057 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:46.451058 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:46.451064 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:46.452136 (+ 1072us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:46.452424 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:46.452424 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:46.452425 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:46.452469 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:46.452618 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:46.452618 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:46.452618 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:46.452748 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:46.452824 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:46.452888 (+ 64us) server_negotiation.cc:299] Negotiation successful
0924 02:57:46.452949 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260924 02:57:46.453215 3432 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:36698
I20260924 02:57:46.453681 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 3494
I20260924 02:57:46.457844 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 3628
I20260924 02:57:46.461563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 3762
I20260924 02:57:46.464955 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 3400
2026-09-24T02:57:46Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1734 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:57:46 dist-test-slave-sb6p krb5kdc[3903](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:57:46 dist-test-slave-sb6p krb5kdc[3903](info): set up 2 sockets
Sep 24 02:57:46 dist-test-slave-sb6p krb5kdc[3903](info): commencing operation
krb5kdc: starting...
W20260924 02:57:47.414031 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:57:47 dist-test-slave-sb6p krb5kdc[3903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:57:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:57:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 02:57:47.521268 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35591
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:47.597839 3916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:47.598037 3916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:47.598064 3916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:47.599213 3916 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:47.599252 3916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:47.599269 3916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:47.599287 3916 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:47.599300 3916 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:57:47.600713 3916 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3916
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:47.600956 3916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:47.601150 3916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 02:57:47.605702 3916 server_base.cc:1034] running on GCE node
W20260924 02:57:47.605645 3921 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:47.605631 3924 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:47.605633 3922 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:47.606115 3916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:47.606382 3916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:47.607519 3916 hybrid_clock.cc:648] HybridClock initialized: now 1790218667607496 us; error 40 us; skew 500 ppm
Sep 24 02:57:47 dist-test-slave-sb6p krb5kdc[3903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:47.609819 3916 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:57:47.610210 3916 webserver.cc:460] Webserver started at http://127.0.98.62:34057/ using document root <none> and password file <none>
I20260924 02:57:47.610384 3916 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:47.610422 3916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:47.610508 3916 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:47.611509 3916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "479d24529f7c47d6945f42b7dd14bd32"
format_stamp: "Formatted at 2026-09-24 02:57:47 on dist-test-slave-sb6p"
server_key: "06c58a62cabc236b8b32eaa1a1337441"
server_key_iv: "84a16e2b9d73799934fe093e75a3b8b2"
server_key_version: "encryptionkey@0"
I20260924 02:57:47.611791 3916 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "479d24529f7c47d6945f42b7dd14bd32"
format_stamp: "Formatted at 2026-09-24 02:57:47 on dist-test-slave-sb6p"
server_key: "06c58a62cabc236b8b32eaa1a1337441"
server_key_iv: "84a16e2b9d73799934fe093e75a3b8b2"
server_key_version: "encryptionkey@0"
I20260924 02:57:47.612888 3916 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 02:57:47.613467 3931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:47.613618 3916 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:47.613695 3916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "479d24529f7c47d6945f42b7dd14bd32"
format_stamp: "Formatted at 2026-09-24 02:57:47 on dist-test-slave-sb6p"
server_key: "06c58a62cabc236b8b32eaa1a1337441"
server_key_iv: "84a16e2b9d73799934fe093e75a3b8b2"
server_key_version: "encryptionkey@0"
I20260924 02:57:47.613747 3916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:47.630584 3916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:47.632210 3916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:47.632326 3916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:47.635869 3916 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35591
I20260924 02:57:47.635902 3983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35591 every 8 connection(s)
I20260924 02:57:47.636230 3916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:57:47.637012 3984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:47.639199 3984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32: Bootstrap starting.
I20260924 02:57:47.639730 3984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:47.639988 3984 log.cc:824] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:47.640578 3984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32: No bootstrap required, opened a new log
I20260924 02:57:47.642092 3984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479d24529f7c47d6945f42b7dd14bd32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35591 } }
I20260924 02:57:47.642225 3984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:47.642254 3984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 479d24529f7c47d6945f42b7dd14bd32, State: Initialized, Role: FOLLOWER
I20260924 02:57:47.642379 3984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479d24529f7c47d6945f42b7dd14bd32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35591 } }
I20260924 02:57:47.642438 3984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:57:47.642467 3984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:57:47.642511 3984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:47.643016 3984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479d24529f7c47d6945f42b7dd14bd32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35591 } }
I20260924 02:57:47.643103 3984 leader_election.cc:302] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 479d24529f7c47d6945f42b7dd14bd32; no voters:
I20260924 02:57:47.643252 3984 leader_election.cc:288] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:57:47.643291 3987 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:47.643474 3984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:47.643605 3987 raft_consensus.cc:686] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [term 1 LEADER]: Becoming Leader. State: Replica: 479d24529f7c47d6945f42b7dd14bd32, State: Running, Role: LEADER
I20260924 02:57:47.643695 3987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479d24529f7c47d6945f42b7dd14bd32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35591 } }
I20260924 02:57:47.644060 3988 sys_catalog.cc:455] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "479d24529f7c47d6945f42b7dd14bd32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479d24529f7c47d6945f42b7dd14bd32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35591 } } }
I20260924 02:57:47.644073 3989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 479d24529f7c47d6945f42b7dd14bd32. Latest consensus state: current_term: 1 leader_uuid: "479d24529f7c47d6945f42b7dd14bd32" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "479d24529f7c47d6945f42b7dd14bd32" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35591 } } }
I20260924 02:57:47.644236 3988 sys_catalog.cc:458] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:47.644265 3989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:47.644579 3997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:47.645056 3997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:57:47.645243 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 3916
I20260924 02:57:47.645351 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:57:47.645530 392 external_mini_cluster.cc:1428] Setting key 2cefa048e0960941a118c08b8b195e6b
I20260924 02:57:47.646449 3997 catalog_manager.cc:1347] Generated new cluster ID: da91d27326f54053908223a2d75e85dc
I20260924 02:57:47.646497 3997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 02:57:47 dist-test-slave-sb6p krb5kdc[3903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218667, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:47.653537 3997 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:57:47.654740 3997 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:47.656348 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:47.645957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51830 (local address 127.0.98.62:35591)
0924 02:57:47.646125 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:47.646135 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:47.646238 (+ 103us) server_negotiation.cc:407] Connection header received
0924 02:57:47.646388 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:47.646396 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:47.646452 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:47.646598 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:47.647387 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:47.648829 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:47.649834 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:47.650475 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:47.650638 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:47.653398 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:47.653403 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:47.653403 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:47.653415 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:47.654942 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:47.655245 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:47.655248 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:47.655248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:47.655352 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:47.655527 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:47.655528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:47.655528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:47.655743 (+ 215us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:47.655879 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:47.655996 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 02:57:47.656142 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":73,"threads_started":1}
I20260924 02:57:47.657948 3997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 479d24529f7c47d6945f42b7dd14bd32: Generated new TSK 0
I20260924 02:57:47.658102 3997 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 02:57:47.683439 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35591
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:47.761260 4010 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:47.761457 4010 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:47.761487 4010 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:47.762633 4010 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:47.762670 4010 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:47.762705 4010 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:57:47.764110 4010 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4010
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:47.764323 4010 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:47.764516 4010 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:47.769594 4018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:47.769655 4016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:47.769701 4010 server_base.cc:1034] running on GCE node
W20260924 02:57:47.769603 4015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:47.769977 4010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:47.770183 4010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:47.771304 4010 hybrid_clock.cc:648] HybridClock initialized: now 1790218667771301 us; error 30 us; skew 500 ppm
Sep 24 02:57:47 dist-test-slave-sb6p krb5kdc[3903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:47.773617 4010 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 02:57:47.774004 4010 webserver.cc:460] Webserver started at http://127.0.98.1:46097/ using document root <none> and password file <none>
I20260924 02:57:47.774190 4010 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:47.774237 4010 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:47.774364 4010 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:47.775408 4010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "f13d7789acc540bd9709710fef434d81"
format_stamp: "Formatted at 2026-09-24 02:57:47 on dist-test-slave-sb6p"
server_key: "680fc61870a90c905fa9b81233085850"
server_key_iv: "ddae8732cb0941c41fb4c3cfbc699559"
server_key_version: "encryptionkey@0"
I20260924 02:57:47.775730 4010 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "f13d7789acc540bd9709710fef434d81"
format_stamp: "Formatted at 2026-09-24 02:57:47 on dist-test-slave-sb6p"
server_key: "680fc61870a90c905fa9b81233085850"
server_key_iv: "ddae8732cb0941c41fb4c3cfbc699559"
server_key_version: "encryptionkey@0"
I20260924 02:57:47.776939 4010 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:57:47.777587 4025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:47.777765 4010 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:47.777840 4010 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "f13d7789acc540bd9709710fef434d81"
format_stamp: "Formatted at 2026-09-24 02:57:47 on dist-test-slave-sb6p"
server_key: "680fc61870a90c905fa9b81233085850"
server_key_iv: "ddae8732cb0941c41fb4c3cfbc699559"
server_key_version: "encryptionkey@0"
I20260924 02:57:47.777906 4010 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:47.787441 4010 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:47.789398 4010 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:47.789521 4010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:47.789714 4010 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:47.789999 4010 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:47.790038 4010 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:47.790064 4010 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:47.790088 4010 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:47.796411 4010 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35499
I20260924 02:57:47.796473 4138 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35499 every 8 connection(s)
I20260924 02:57:47.796779 4010 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:57:47.798026 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 4010
I20260924 02:57:47.798130 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:57:47.798319 392 external_mini_cluster.cc:1428] Setting key 4225ec325a8326ba758392381922727a
Sep 24 02:57:47 dist-test-slave-sb6p krb5kdc[3903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 02:57:47.806187 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:47.797727 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:32925 (local address 127.0.98.62:35591)
0924 02:57:47.797862 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:47.797864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:47.797873 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:57:47.798922 (+ 1049us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:47.798923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:47.798936 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:47.798985 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:47.799739 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:47.800252 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:47.801216 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:47.801430 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:47.801486 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:47.803673 (+ 2187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:47.803674 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:47.803675 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:47.803681 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:47.804860 (+ 1179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:47.805328 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:47.805329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:47.805329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:47.805376 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:47.805607 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:47.805607 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:47.805607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:47.805715 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:47.805763 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:47.806049 (+ 286us) server_negotiation.cc:299] Negotiation successful
0924 02:57:47.806124 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260924 02:57:47.806346 4141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:47.797467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35591 (local address 127.0.98.1:32925)
0924 02:57:47.797623 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:47.797693 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:47.798845 (+ 1152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:47.799058 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:47.799062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:47.799384 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:47.799655 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:47.799660 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:47.800345 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:47.800346 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:47.801140 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:47.801142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:47.801497 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:47.801498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:47.801621 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:47.802050 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:47.802058 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:47.803586 (+ 1528us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:47.804939 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:47.804945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:47.804947 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:47.805261 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:47.805428 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:47.805428 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:47.805429 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:47.805553 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:47.805760 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:47.805766 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:47.805931 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 02:57:47.806033 (+ 102us) client_negotiation.cc:240] Negotiation successful
0924 02:57:47.806161 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260924 02:57:47.806800 4139 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35591
I20260924 02:57:47.806877 4139 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:47.807011 4139 heartbeater.cc:511] Master 127.0.98.62:35591 requested a full tablet report, sending...
I20260924 02:57:47.807478 3948 ts_manager.cc:194] Registered new tserver with Master: f13d7789acc540bd9709710fef434d81 (127.0.98.1:35499)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
I20260924 02:57:47.808139 3948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:32925
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 02:57:47.820851 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35591
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:47.897681 4144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:47.897886 4144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:47.897908 4144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:47.899025 4144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:47.899060 4144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:47.899083 4144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:57:47.900492 4144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:47.900678 4144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:47.900851 4144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:47.905457 4150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:47.905422 4152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:47.905480 4144 server_base.cc:1034] running on GCE node
W20260924 02:57:47.905437 4149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:47.905807 4144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:47.906015 4144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:47.907140 4144 hybrid_clock.cc:648] HybridClock initialized: now 1790218667907133 us; error 33 us; skew 500 ppm
Sep 24 02:57:47 dist-test-slave-sb6p krb5kdc[3903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:47.909411 4144 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 02:57:47.909795 4144 webserver.cc:460] Webserver started at http://127.0.98.2:33201/ using document root <none> and password file <none>
I20260924 02:57:47.909981 4144 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:47.910027 4144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:47.910145 4144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:47.911151 4144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "f94ef6a28a53473c9b45d23a41e2226d"
format_stamp: "Formatted at 2026-09-24 02:57:47 on dist-test-slave-sb6p"
server_key: "30bc6cdd30cd2e813008fc6e9db9cdf7"
server_key_iv: "d7ea67e1c8ee41e5a578d0884673798e"
server_key_version: "encryptionkey@0"
I20260924 02:57:47.911459 4144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f94ef6a28a53473c9b45d23a41e2226d"
format_stamp: "Formatted at 2026-09-24 02:57:47 on dist-test-slave-sb6p"
server_key: "30bc6cdd30cd2e813008fc6e9db9cdf7"
server_key_iv: "d7ea67e1c8ee41e5a578d0884673798e"
server_key_version: "encryptionkey@0"
I20260924 02:57:47.912588 4144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:57:47.913160 4159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:47.913300 4144 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:47.913376 4144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "f94ef6a28a53473c9b45d23a41e2226d"
format_stamp: "Formatted at 2026-09-24 02:57:47 on dist-test-slave-sb6p"
server_key: "30bc6cdd30cd2e813008fc6e9db9cdf7"
server_key_iv: "d7ea67e1c8ee41e5a578d0884673798e"
server_key_version: "encryptionkey@0"
I20260924 02:57:47.913432 4144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:47.929634 4144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:47.931653 4144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:47.931778 4144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:47.931965 4144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:47.932233 4144 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:47.932277 4144 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:47.932307 4144 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:47.932329 4144 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:47.937402 4144 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43869
I20260924 02:57:47.937455 4272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43869 every 8 connection(s)
I20260924 02:57:47.937714 4144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 02:57:47 dist-test-slave-sb6p krb5kdc[3903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:47.944794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 4144
I20260924 02:57:47.944906 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:57:47.945086 392 external_mini_cluster.cc:1428] Setting key 1a9646f71ae704ab1a22d644b793e7dd
I20260924 02:57:47.946415 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:47.938320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40161 (local address 127.0.98.62:35591)
0924 02:57:47.938370 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:47.938371 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:47.938567 (+ 196us) server_negotiation.cc:407] Connection header received
0924 02:57:47.939474 (+ 907us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:47.939475 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:47.939487 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:47.939534 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:47.940115 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:47.940597 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:47.941534 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:47.941714 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:47.941767 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:47.943682 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:47.943683 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:47.943684 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:47.943691 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:47.944953 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:47.945468 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:47.945469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:47.945470 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:47.945520 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:47.945776 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:47.945776 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:47.945777 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:47.945910 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:47.945974 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:47.946245 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 02:57:47.946329 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 02:57:47.946508 4275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:47.938295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35591 (local address 127.0.98.2:40161)
0924 02:57:47.938449 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:47.938482 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:47.939378 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:47.939616 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:47.939620 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:47.939860 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:47.940035 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:47.940039 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:47.940673 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:47.940674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:47.941458 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:47.941459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:47.941783 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:47.941784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:47.941895 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:47.942279 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:47.942287 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:47.943607 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:47.945045 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:47.945052 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:47.945053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:47.945408 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:47.945572 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:47.945573 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:47.945574 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:47.945709 (+ 135us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:47.945968 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:47.945972 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:47.946095 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 02:57:47.946219 (+ 124us) client_negotiation.cc:240] Negotiation successful
0924 02:57:47.946340 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":75,"threads_started":1}
I20260924 02:57:47.947059 4273 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35591
I20260924 02:57:47.947177 4273 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:47.947356 4273 heartbeater.cc:511] Master 127.0.98.62:35591 requested a full tablet report, sending...
I20260924 02:57:47.947678 3948 ts_manager.cc:194] Registered new tserver with Master: f94ef6a28a53473c9b45d23a41e2226d (127.0.98.2:43869)
I20260924 02:57:47.948048 3948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40161
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 02:57:47.966825 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35591
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:48.044322 4278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:48.044556 4278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:48.044581 4278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:48.045676 4278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:48.045709 4278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:48.045733 4278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:57:48.047159 4278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:35591
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:48.047333 4278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:48.047497 4278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:48.052057 4286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:48.052074 4284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:48.052158 4278 server_base.cc:1034] running on GCE node
W20260924 02:57:48.052248 4283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:48.052428 4278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:48.052644 4278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:48.053789 4278 hybrid_clock.cc:648] HybridClock initialized: now 1790218668053775 us; error 27 us; skew 500 ppm
Sep 24 02:57:48 dist-test-slave-sb6p krb5kdc[3903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:48.056034 4278 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 02:57:48.056464 4278 webserver.cc:460] Webserver started at http://127.0.98.3:43211/ using document root <none> and password file <none>
I20260924 02:57:48.056670 4278 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:48.056716 4278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:48.056842 4278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:48.057844 4278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "e719fe46b953439aad0b8fc1fa45dc95"
format_stamp: "Formatted at 2026-09-24 02:57:48 on dist-test-slave-sb6p"
server_key: "7f0aedd4df923e1dc99b32eebc45e0c6"
server_key_iv: "d1df4f7903381a5376ef7dcb5c3b9c7d"
server_key_version: "encryptionkey@0"
I20260924 02:57:48.058149 4278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "e719fe46b953439aad0b8fc1fa45dc95"
format_stamp: "Formatted at 2026-09-24 02:57:48 on dist-test-slave-sb6p"
server_key: "7f0aedd4df923e1dc99b32eebc45e0c6"
server_key_iv: "d1df4f7903381a5376ef7dcb5c3b9c7d"
server_key_version: "encryptionkey@0"
I20260924 02:57:48.060083 4278 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.003s
I20260924 02:57:48.060927 4293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:48.061095 4278 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:48.061190 4278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "e719fe46b953439aad0b8fc1fa45dc95"
format_stamp: "Formatted at 2026-09-24 02:57:48 on dist-test-slave-sb6p"
server_key: "7f0aedd4df923e1dc99b32eebc45e0c6"
server_key_iv: "d1df4f7903381a5376ef7dcb5c3b9c7d"
server_key_version: "encryptionkey@0"
I20260924 02:57:48.061259 4278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:48.075431 4278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:48.077487 4278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:48.077601 4278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:48.077798 4278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:48.078040 4278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:48.078068 4278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:48.078088 4278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:48.078105 4278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:48.083564 4278 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43217
I20260924 02:57:48.083608 4406 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43217 every 8 connection(s)
I20260924 02:57:48.083910 4278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 02:57:48 dist-test-slave-sb6p krb5kdc[3903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218668, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:48.090796 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 4278
I20260924 02:57:48.090896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:57:48.091087 392 external_mini_cluster.cc:1428] Setting key 5520c7fef5b81437e3b118c4966fcaec
I20260924 02:57:48.092553 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:48.084593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59141 (local address 127.0.98.62:35591)
0924 02:57:48.084646 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:48.084648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:48.084827 (+ 179us) server_negotiation.cc:407] Connection header received
0924 02:57:48.085748 (+ 921us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:48.085750 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:48.085777 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:48.085832 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:48.086493 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:48.087023 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:48.087981 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:48.088169 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:48.088223 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:48.090086 (+ 1863us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:48.090087 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:48.090087 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:48.090094 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:48.091413 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:48.091837 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:48.091837 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:48.091838 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:48.091894 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:48.092087 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:48.092087 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:48.092088 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:48.092181 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:48.092225 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:48.092429 (+ 204us) server_negotiation.cc:299] Negotiation successful
0924 02:57:48.092491 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 02:57:48.092705 4409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:48.084567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35591 (local address 127.0.98.3:59141)
0924 02:57:48.084708 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:48.084744 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:48.085642 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:48.085916 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:48.085920 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:48.086171 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:48.086399 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:48.086404 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:48.087102 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:48.087103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:48.087898 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:48.087900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:48.088233 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:48.088234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:48.088354 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:48.088750 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:48.088758 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:48.090003 (+ 1245us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:48.091490 (+ 1487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:48.091496 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:48.091498 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:48.091760 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:48.091948 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:48.091949 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:48.091949 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:48.092042 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:48.092221 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:48.092224 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:48.092337 (+ 113us) client_negotiation.cc:769] Sending connection context
0924 02:57:48.092412 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 02:57:48.092524 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260924 02:57:48.093087 4407 heartbeater.cc:347] Connected to a master server at 127.0.98.62:35591
I20260924 02:57:48.093174 4407 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:48.093341 4407 heartbeater.cc:511] Master 127.0.98.62:35591 requested a full tablet report, sending...
I20260924 02:57:48.093647 3948 ts_manager.cc:194] Registered new tserver with Master: e719fe46b953439aad0b8fc1fa45dc95 (127.0.98.3:43217)
I20260924 02:57:48.093986 3948 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:59141
I20260924 02:57:48.102579 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 02:57:48 dist-test-slave-sb6p krb5kdc[3903](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 02:57:48 dist-test-slave-sb6p krb5kdc[3903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 02:57:48.122951 4414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:48.116140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53454 (local address 127.0.98.1:35499)
0924 02:57:48.116286 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:48.116291 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:48.116303 (+ 12us) server_negotiation.cc:407] Connection header received
0924 02:57:48.116332 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:48.116334 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:48.116348 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:48.116413 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:48.116986 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:48.117688 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:48.118371 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:48.118715 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:48.118775 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:48.120767 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:48.120774 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:48.120774 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:48.120780 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:48.122017 (+ 1237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:48.122290 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:48.122292 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:48.122292 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:48.122351 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:48.122469 (+ 118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:48.122470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:48.122470 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:48.122610 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:48.122740 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:48.122811 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 02:57:48.122872 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
Sep 24 02:57:48 dist-test-slave-sb6p krb5kdc[3903](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218668, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:48.129779 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:48.124066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51838 (local address 127.0.98.62:35591)
0924 02:57:48.124116 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:48.124117 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:48.124136 (+ 19us) server_negotiation.cc:407] Connection header received
0924 02:57:48.124186 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:48.124187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:48.124196 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:48.124236 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:48.124608 (+ 372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:48.125076 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:48.125777 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:48.125989 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:48.126022 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:48.127959 (+ 1937us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:48.127960 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:48.127960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:48.127967 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:48.128974 (+ 1007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:48.129230 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:48.129231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:48.129231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:48.129274 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:48.129390 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:48.129390 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:48.129391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:48.129510 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:48.129577 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:48.129644 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 02:57:48.129697 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
W20260924 02:57:48.129993 3948 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51838
I20260924 02:57:48.130306 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 4010
I20260924 02:57:48.134403 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 4144
I20260924 02:57:48.137637 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 4278
I20260924 02:57:48.141100 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 3916
2026-09-24T02:57:48Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1675 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:57:48 dist-test-slave-sb6p krb5kdc[4419](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:57:48 dist-test-slave-sb6p krb5kdc[4419](info): set up 2 sockets
Sep 24 02:57:48 dist-test-slave-sb6p krb5kdc[4419](info): commencing operation
krb5kdc: starting...
W20260924 02:57:49.094190 392 mini_kdc.cc:121] Time spent starting KDC: real 0.935s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:57:49 dist-test-slave-sb6p krb5kdc[4419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:57:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:57:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 02:57:49.201174 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33157
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33157
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:49.279238 4432 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:49.279430 4432 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:49.279451 4432 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:49.280580 4432 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:49.280618 4432 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:49.280632 4432 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:49.280652 4432 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:49.280676 4432 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:57:49.282133 4432 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33157
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33157
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4432
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:49.282348 4432 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:49.282575 4432 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:49.286942 4437 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:49.287075 4432 server_base.cc:1034] running on GCE node
W20260924 02:57:49.286957 4438 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:49.286963 4440 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:49.287395 4432 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:49.287657 4432 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:49.288785 4432 hybrid_clock.cc:648] HybridClock initialized: now 1790218669288767 us; error 34 us; skew 500 ppm
Sep 24 02:57:49 dist-test-slave-sb6p krb5kdc[4419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:49.291096 4432 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:57:49.291393 4432 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:49.291437 4432 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:49.291524 4432 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:49.292490 4432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "e272c8d5c83640b9a8002fc448f55e29"
format_stamp: "Formatted at 2026-09-24 02:57:49 on dist-test-slave-sb6p"
server_key: "b1f9b9b1aed2d46d8e8053c7a02f5da6"
server_key_iv: "9f073fb4988e0edf33b203d2c15af02e"
server_key_version: "encryptionkey@0"
I20260924 02:57:49.292765 4432 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "e272c8d5c83640b9a8002fc448f55e29"
format_stamp: "Formatted at 2026-09-24 02:57:49 on dist-test-slave-sb6p"
server_key: "b1f9b9b1aed2d46d8e8053c7a02f5da6"
server_key_iv: "9f073fb4988e0edf33b203d2c15af02e"
server_key_version: "encryptionkey@0"
I20260924 02:57:49.293877 4432 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:57:49.294461 4446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:49.294625 4432 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:49.294704 4432 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "e272c8d5c83640b9a8002fc448f55e29"
format_stamp: "Formatted at 2026-09-24 02:57:49 on dist-test-slave-sb6p"
server_key: "b1f9b9b1aed2d46d8e8053c7a02f5da6"
server_key_iv: "9f073fb4988e0edf33b203d2c15af02e"
server_key_version: "encryptionkey@0"
I20260924 02:57:49.294767 4432 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:49.316550 4432 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:49.318354 4432 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:49.318480 4432 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:49.322063 4432 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33157
I20260924 02:57:49.322091 4498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33157 every 8 connection(s)
I20260924 02:57:49.322368 4432 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:57:49.323370 4499 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:49.325052 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 4432
I20260924 02:57:49.325143 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:57:49.325300 392 external_mini_cluster.cc:1428] Setting key 9bd3939b84f8fe47a4aa79ed8a05778c
I20260924 02:57:49.325865 4499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29: Bootstrap starting.
I20260924 02:57:49.326669 4499 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:49.326987 4499 log.cc:824] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:49.327663 4499 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29: No bootstrap required, opened a new log
I20260924 02:57:49.329810 4499 raft_consensus.cc:348] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e272c8d5c83640b9a8002fc448f55e29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33157 } }
I20260924 02:57:49.329977 4499 raft_consensus.cc:374] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:49.330009 4499 raft_consensus.cc:729] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e272c8d5c83640b9a8002fc448f55e29, State: Initialized, Role: FOLLOWER
I20260924 02:57:49.330142 4499 consensus_queue.cc:260] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e272c8d5c83640b9a8002fc448f55e29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33157 } }
I20260924 02:57:49.330225 4499 raft_consensus.cc:388] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:57:49.330255 4499 raft_consensus.cc:482] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:57:49.330324 4499 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:49.330896 4499 raft_consensus.cc:504] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e272c8d5c83640b9a8002fc448f55e29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33157 } }
I20260924 02:57:49.330982 4499 leader_election.cc:302] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e272c8d5c83640b9a8002fc448f55e29; no voters:
I20260924 02:57:49.331135 4499 leader_election.cc:288] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:57:49.331187 4504 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:49.331369 4499 sys_catalog.cc:564] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:49.331401 4504 raft_consensus.cc:686] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [term 1 LEADER]: Becoming Leader. State: Replica: e272c8d5c83640b9a8002fc448f55e29, State: Running, Role: LEADER
I20260924 02:57:49.331559 4504 consensus_queue.cc:237] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e272c8d5c83640b9a8002fc448f55e29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33157 } }
I20260924 02:57:49.331933 4504 sys_catalog.cc:455] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e272c8d5c83640b9a8002fc448f55e29. Latest consensus state: current_term: 1 leader_uuid: "e272c8d5c83640b9a8002fc448f55e29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e272c8d5c83640b9a8002fc448f55e29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33157 } } }
I20260924 02:57:49.332006 4504 sys_catalog.cc:458] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:49.331990 4506 sys_catalog.cc:455] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e272c8d5c83640b9a8002fc448f55e29" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e272c8d5c83640b9a8002fc448f55e29" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33157 } } }
I20260924 02:57:49.332075 4506 sys_catalog.cc:458] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29 [sys.catalog]: This master's current role is: LEADER
Sep 24 02:57:49 dist-test-slave-sb6p krb5kdc[4419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218669, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
W20260924 02:57:49.333307 4519 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 02:57:49.333362 4519 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 02:57:49.333454 4512 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:49.333649 4512 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:57:49.335180 4512 catalog_manager.cc:1347] Generated new cluster ID: 7ba9b8fa5c4f4f1692261b594754ae1a
I20260924 02:57:49.335232 4512 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:57:49.335996 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.325789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46012 (local address 127.0.98.62:33157)
0924 02:57:49.325968 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:49.325978 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:49.326011 (+ 33us) server_negotiation.cc:407] Connection header received
0924 02:57:49.326614 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:49.326624 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:49.326692 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:49.326850 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:49.327428 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.328785 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.329509 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.330117 (+ 608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.330216 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.332825 (+ 2609us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:49.332829 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:49.332830 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:49.332842 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:49.334473 (+ 1631us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:49.334885 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:49.334887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:49.334888 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:49.334984 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:49.335169 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:49.335170 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:49.335170 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:49.335428 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:49.335556 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:49.335665 (+ 109us) server_negotiation.cc:299] Negotiation successful
0924 02:57:49.335790 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":77,"threads_started":1}
I20260924 02:57:49.340078 4512 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:57:49.340574 4512 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:49.345877 4512 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29: Generated new TSK 0
I20260924 02:57:49.346028 4512 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 02:57:49.379359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33157
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:49.457916 4525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:49.458119 4525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:49.458166 4525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:49.459375 4525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:49.459421 4525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:49.459458 4525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:57:49.460974 4525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4525
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:49.461216 4525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:49.461421 4525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:49.466137 4530 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:49.466236 4533 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:49.466351 4525 server_base.cc:1034] running on GCE node
W20260924 02:57:49.466405 4531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:49.466638 4525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:49.466872 4525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:49.468036 4525 hybrid_clock.cc:648] HybridClock initialized: now 1790218669468020 us; error 27 us; skew 500 ppm
Sep 24 02:57:49 dist-test-slave-sb6p krb5kdc[4419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:49.470415 4525 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 02:57:49.470764 4525 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:49.470811 4525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:49.470902 4525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:49.471860 4525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "b96e743a5ef048f3ad163e3796dbc586"
format_stamp: "Formatted at 2026-09-24 02:57:49 on dist-test-slave-sb6p"
server_key: "91ce8e9b34156b7e50bfc04aed89b5db"
server_key_iv: "269a06ea04d252463773eb325290ba24"
server_key_version: "encryptionkey@0"
I20260924 02:57:49.472136 4525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "b96e743a5ef048f3ad163e3796dbc586"
format_stamp: "Formatted at 2026-09-24 02:57:49 on dist-test-slave-sb6p"
server_key: "91ce8e9b34156b7e50bfc04aed89b5db"
server_key_iv: "269a06ea04d252463773eb325290ba24"
server_key_version: "encryptionkey@0"
I20260924 02:57:49.473213 4525 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 02:57:49.473861 4539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:49.474004 4525 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:49.474076 4525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "b96e743a5ef048f3ad163e3796dbc586"
format_stamp: "Formatted at 2026-09-24 02:57:49 on dist-test-slave-sb6p"
server_key: "91ce8e9b34156b7e50bfc04aed89b5db"
server_key_iv: "269a06ea04d252463773eb325290ba24"
server_key_version: "encryptionkey@0"
I20260924 02:57:49.474143 4525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:49.492700 4525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:49.494406 4525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:49.494582 4525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:49.494793 4525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:49.495074 4525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:49.495105 4525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:49.495136 4525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:49.495158 4525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:49.500569 4525 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37517
I20260924 02:57:49.500622 4652 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37517 every 8 connection(s)
I20260924 02:57:49.500855 4525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:57:49.503932 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 4525
I20260924 02:57:49.504015 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:57:49.504170 392 external_mini_cluster.cc:1428] Setting key bbe4a4b11e3f41547a95ea60c7a39ff1
Sep 24 02:57:49 dist-test-slave-sb6p krb5kdc[4419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:49.509897 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.501476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36053 (local address 127.0.98.62:33157)
0924 02:57:49.501524 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:49.501526 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:49.501726 (+ 200us) server_negotiation.cc:407] Connection header received
0924 02:57:49.502205 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:49.502206 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:49.502215 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:49.502263 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:49.503164 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.503671 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.504808 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.505090 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.505155 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.507343 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:49.507345 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:49.507345 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:49.507353 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:49.508624 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:49.509006 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:49.509008 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:49.509008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:49.509061 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:49.509284 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:49.509285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:49.509285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:49.509405 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:49.509461 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:49.509712 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 02:57:49.509803 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 02:57:49.510083 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.501451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33157 (local address 127.0.98.1:36053)
0924 02:57:49.501611 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:49.501648 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:49.502116 (+ 468us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:49.502325 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:49.502328 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:49.502601 (+ 273us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:49.503086 (+ 485us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:49.503091 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.503735 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.503736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:49.504724 (+ 988us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:49.504727 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.505159 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.505160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:49.505260 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.505727 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:49.505735 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:49.507254 (+ 1519us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:49.508702 (+ 1448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:49.508710 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:49.508713 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:49.508938 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:49.509116 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:49.509117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:49.509117 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:49.509226 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:49.509459 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:49.509465 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:49.509597 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 02:57:49.509685 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 02:57:49.509869 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260924 02:57:49.510531 4653 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33157
I20260924 02:57:49.510625 4653 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:49.510769 4653 heartbeater.cc:511] Master 127.0.98.62:33157 requested a full tablet report, sending...
I20260924 02:57:49.511327 4463 ts_manager.cc:194] Registered new tserver with Master: b96e743a5ef048f3ad163e3796dbc586 (127.0.98.1:37517)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 02:57:49.512185 4463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36053
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 02:57:49.527299 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33157
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:49.604951 4658 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:49.605134 4658 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:49.605155 4658 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:49.606268 4658 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:49.606307 4658 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:49.606333 4658 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:57:49.607812 4658 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4658
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:49.607993 4658 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:49.608165 4658 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:49.612710 4666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:49.612685 4663 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:49.612756 4664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:49.612833 4658 server_base.cc:1034] running on GCE node
I20260924 02:57:49.613155 4658 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:49.613376 4658 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:49.614504 4658 hybrid_clock.cc:648] HybridClock initialized: now 1790218669614493 us; error 27 us; skew 500 ppm
Sep 24 02:57:49 dist-test-slave-sb6p krb5kdc[4419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:49.616760 4658 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 02:57:49.617058 4658 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:49.617103 4658 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:49.617190 4658 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:49.618149 4658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "5da262eb236f4ac7b55d2b0d7fc11d82"
format_stamp: "Formatted at 2026-09-24 02:57:49 on dist-test-slave-sb6p"
server_key: "c324498ece733017c4bd699e42e28403"
server_key_iv: "8df6429d9ba0c378f47d5747de0dc247"
server_key_version: "encryptionkey@0"
I20260924 02:57:49.618423 4658 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "5da262eb236f4ac7b55d2b0d7fc11d82"
format_stamp: "Formatted at 2026-09-24 02:57:49 on dist-test-slave-sb6p"
server_key: "c324498ece733017c4bd699e42e28403"
server_key_iv: "8df6429d9ba0c378f47d5747de0dc247"
server_key_version: "encryptionkey@0"
I20260924 02:57:49.619525 4658 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 02:57:49.620122 4672 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:49.620271 4658 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:49.620358 4658 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "5da262eb236f4ac7b55d2b0d7fc11d82"
format_stamp: "Formatted at 2026-09-24 02:57:49 on dist-test-slave-sb6p"
server_key: "c324498ece733017c4bd699e42e28403"
server_key_iv: "8df6429d9ba0c378f47d5747de0dc247"
server_key_version: "encryptionkey@0"
I20260924 02:57:49.620424 4658 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:49.631927 4658 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:49.633956 4658 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:49.634078 4658 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:49.634279 4658 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:49.634608 4658 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:49.634644 4658 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:49.634665 4658 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:49.634688 4658 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:49.639963 4658 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:33251
I20260924 02:57:49.640014 4785 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:33251 every 8 connection(s)
I20260924 02:57:49.640244 4658 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 02:57:49.641491 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 4658
I20260924 02:57:49.641587 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:57:49.641752 392 external_mini_cluster.cc:1428] Setting key e90e63a4e4591a3dee9743b468c8ae29
Sep 24 02:57:49 dist-test-slave-sb6p krb5kdc[4419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 02:57:49.649616 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.640840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58571 (local address 127.0.98.62:33157)
0924 02:57:49.640881 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:49.640883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:49.641084 (+ 201us) server_negotiation.cc:407] Connection header received
0924 02:57:49.641537 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:49.641539 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:49.641551 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:49.641618 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:49.642664 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.643356 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.644663 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.644955 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.645025 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.647048 (+ 2023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:49.647049 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:49.647049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:49.647056 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:49.648225 (+ 1169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:49.648663 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:49.648663 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:49.648664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:49.648707 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:49.648997 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:49.648998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:49.648998 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:49.649113 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:49.649182 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:49.649489 (+ 307us) server_negotiation.cc:299] Negotiation successful
0924 02:57:49.649551 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 02:57:49.649830 4788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.640843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33157 (local address 127.0.98.2:58571)
0924 02:57:49.640976 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:49.641009 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:49.641456 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:49.641712 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:49.641715 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:49.641995 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:49.642582 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:49.642588 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.643430 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.643431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:49.644584 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:49.644587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.645025 (+ 438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.645026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:49.645121 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.645541 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:49.645549 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:49.646957 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:49.648305 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:49.648311 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:49.648314 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:49.648592 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:49.648765 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:49.648766 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:49.648767 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:49.648905 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:49.649182 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:49.649187 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:49.649352 (+ 165us) client_negotiation.cc:769] Sending connection context
0924 02:57:49.649467 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 02:57:49.649621 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260924 02:57:49.650342 4786 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33157
I20260924 02:57:49.650436 4786 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:49.650596 4786 heartbeater.cc:511] Master 127.0.98.62:33157 requested a full tablet report, sending...
I20260924 02:57:49.650941 4463 ts_manager.cc:194] Registered new tserver with Master: 5da262eb236f4ac7b55d2b0d7fc11d82 (127.0.98.2:33251)
I20260924 02:57:49.651332 4463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58571
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 02:57:49.665369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33157
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35605
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:49.743260 4791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:49.743469 4791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:49.743490 4791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:49.744583 4791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:49.744616 4791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:49.744640 4791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:57:49.746101 4791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35605
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33157
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4791
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:49.746323 4791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:49.746505 4791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:49.751112 4796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:49.751178 4791 server_base.cc:1034] running on GCE node
W20260924 02:57:49.751119 4799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:49.751124 4797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:49.751538 4791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:49.751749 4791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:49.752882 4791 hybrid_clock.cc:648] HybridClock initialized: now 1790218669752867 us; error 26 us; skew 500 ppm
Sep 24 02:57:49 dist-test-slave-sb6p krb5kdc[4419](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:49.755198 4791 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 02:57:49.755501 4791 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:49.755544 4791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:49.755650 4791 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:49.756652 4791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "72fd550323c045a48f7e0b83b47a778c"
format_stamp: "Formatted at 2026-09-24 02:57:49 on dist-test-slave-sb6p"
server_key: "979e024fa4b99ec72e5fd75bbb7e38c9"
server_key_iv: "98539415452b66fd9d0b09b59f7e70f4"
server_key_version: "encryptionkey@0"
I20260924 02:57:49.756948 4791 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "72fd550323c045a48f7e0b83b47a778c"
format_stamp: "Formatted at 2026-09-24 02:57:49 on dist-test-slave-sb6p"
server_key: "979e024fa4b99ec72e5fd75bbb7e38c9"
server_key_iv: "98539415452b66fd9d0b09b59f7e70f4"
server_key_version: "encryptionkey@0"
I20260924 02:57:49.758067 4791 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 02:57:49.758733 4805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:49.758888 4791 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:49.758965 4791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "72fd550323c045a48f7e0b83b47a778c"
format_stamp: "Formatted at 2026-09-24 02:57:49 on dist-test-slave-sb6p"
server_key: "979e024fa4b99ec72e5fd75bbb7e38c9"
server_key_iv: "98539415452b66fd9d0b09b59f7e70f4"
server_key_version: "encryptionkey@0"
I20260924 02:57:49.759017 4791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:49.768415 4791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:49.769989 4791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:49.770087 4791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:49.770242 4791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:49.770480 4791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:49.770509 4791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:49.770563 4791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:49.770586 4791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:49.775935 4791 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38297
I20260924 02:57:49.775965 4918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38297 every 8 connection(s)
I20260924 02:57:49.776211 4791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 02:57:49.779517 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 4791
I20260924 02:57:49.779605 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:57:49.779774 392 external_mini_cluster.cc:1428] Setting key bdb428658e93b4ed0475fd71915412e3
Sep 24 02:57:49 dist-test-slave-sb6p krb5kdc[4419](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218669, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:49.784868 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.776899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48497 (local address 127.0.98.62:33157)
0924 02:57:49.776959 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:49.776962 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:49.777190 (+ 228us) server_negotiation.cc:407] Connection header received
0924 02:57:49.777664 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:49.777665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:49.777675 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:49.777730 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:49.778684 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.779203 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.780490 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.780750 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.780804 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.782711 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:49.782712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:49.782713 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:49.782719 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:49.783921 (+ 1202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:49.784214 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:49.784214 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:49.784215 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:49.784257 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:49.784429 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:49.784429 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:49.784430 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:49.784521 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:49.784564 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:49.784752 (+ 188us) server_negotiation.cc:299] Negotiation successful
0924 02:57:49.784805 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 02:57:49.785105 4921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.776867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33157 (local address 127.0.98.3:48497)
0924 02:57:49.777063 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:49.777103 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:49.777568 (+ 465us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:49.777792 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:49.777796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:49.778064 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:49.778598 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:49.778605 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.779304 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.779304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:49.780308 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:49.780311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.780818 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.780819 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:49.780917 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.781368 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:49.781375 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:49.782630 (+ 1255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:49.783978 (+ 1348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:49.783981 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:49.783983 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:49.784166 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:49.784305 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:49.784306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:49.784306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:49.784389 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:49.784560 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:49.784563 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:49.784668 (+ 105us) client_negotiation.cc:769] Sending connection context
0924 02:57:49.784741 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 02:57:49.784850 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260924 02:57:49.785503 4919 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33157
I20260924 02:57:49.785589 4919 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:49.785698 4919 heartbeater.cc:511] Master 127.0.98.62:33157 requested a full tablet report, sending...
I20260924 02:57:49.785997 4463 ts_manager.cc:194] Registered new tserver with Master: 72fd550323c045a48f7e0b83b47a778c (127.0.98.3:38297)
I20260924 02:57:49.786329 4463 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:48497
I20260924 02:57:49.791359 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 02:57:49.799688 4502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.792809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46020 (local address 127.0.98.62:33157)
0924 02:57:49.792880 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:49.792881 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:49.792992 (+ 111us) server_negotiation.cc:407] Connection header received
0924 02:57:49.793073 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:49.793074 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:49.793085 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:49.793162 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:49.793858 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.794593 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.795607 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.795826 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.795882 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.797506 (+ 1624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:49.797507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:49.797507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:49.797513 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:49.798825 (+ 1312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:49.799102 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:49.799103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:49.799103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:49.799145 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:49.799262 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:49.799263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:49.799263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:49.799382 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:49.799450 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:49.799539 (+ 89us) server_negotiation.cc:299] Negotiation successful
0924 02:57:49.799616 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 02:57:49.800933 4463 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46020:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 02:57:49.802392 4463 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:57:49.808415 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.804392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50300 (local address 127.0.98.2:33251)
0924 02:57:49.804796 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:49.804801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:49.804811 (+ 10us) server_negotiation.cc:407] Connection header received
0924 02:57:49.804853 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:49.804855 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:49.804870 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:49.804945 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:49.805108 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.806250 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.807408 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.808044 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.808117 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.808174 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:49.808297 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 02:57:49.808336 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":98,"threads_started":1}
I20260924 02:57:49.808411 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.804311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33251 (local address 127.0.0.1:50300)
0924 02:57:49.804629 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:49.804667 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:49.804726 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:49.804951 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:49.804954 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:49.804956 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:49.805021 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:49.805022 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.806370 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.806371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:49.807338 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:49.807341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.808126 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.808127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:49.808198 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.808211 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 02:57:49.808276 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 02:57:49.808325 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":288,"mutex_wait_us":9,"thread_start_us":64,"threads_started":1}
I20260924 02:57:49.808583 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.804484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35260 (local address 127.0.98.1:37517)
0924 02:57:49.805105 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:49.805108 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:49.805116 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:57:49.805150 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:49.805152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:49.805166 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:49.805232 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:49.805424 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.806103 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.807651 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.808134 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.808196 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.808251 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:49.808457 (+ 206us) server_negotiation.cc:299] Negotiation successful
0924 02:57:49.808498 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":583,"thread_start_us":68,"threads_started":1}
I20260924 02:57:49.808816 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.804596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36168 (local address 127.0.98.3:38297)
0924 02:57:49.804767 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:49.804772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:49.804873 (+ 101us) server_negotiation.cc:407] Connection header received
0924 02:57:49.804948 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:49.804950 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:49.804964 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:49.805034 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:49.805223 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.806263 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.807632 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.808305 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.808364 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.808415 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:49.808721 (+ 306us) server_negotiation.cc:299] Negotiation successful
0924 02:57:49.808760 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260924 02:57:49.808660 4927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.804504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38297 (local address 127.0.0.1:36168)
0924 02:57:49.804827 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:49.804834 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:49.804877 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:49.805119 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:49.805119 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:49.805121 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:49.805159 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:49.805160 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.806349 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.806349 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:49.807554 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:49.807556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.808447 (+ 891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.808448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:49.808508 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.808512 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:57:49.808559 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 02:57:49.808597 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"mutex_wait_us":45,"thread_start_us":76,"threads_started":1}
I20260924 02:57:49.808847 4928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.804321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37517 (local address 127.0.0.1:35260)
0924 02:57:49.804629 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:49.804670 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:49.804726 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:49.805320 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:49.805320 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:49.805322 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:49.805356 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:49.805357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.806206 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.806207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:49.807554 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:49.807556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.808292 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.808293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:49.808343 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.808346 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:49.808389 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 02:57:49.808418 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":60,"thread_start_us":56,"threads_started":1}
I20260924 02:57:49.809528 4853 tablet_service.cc:1467] Processing CreateTablet for tablet ec160828659c41a0b2a26a93050ef767 (DEFAULT_TABLE table=test-table [id=7a45790b69bc465ca23c930fea58b3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:49.809545 4720 tablet_service.cc:1467] Processing CreateTablet for tablet ec160828659c41a0b2a26a93050ef767 (DEFAULT_TABLE table=test-table [id=7a45790b69bc465ca23c930fea58b3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:49.809823 4853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec160828659c41a0b2a26a93050ef767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:49.809826 4720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec160828659c41a0b2a26a93050ef767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:49.812248 4933 tablet_bootstrap.cc:492] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c: Bootstrap starting.
I20260924 02:57:49.812250 4934 tablet_bootstrap.cc:492] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82: Bootstrap starting.
I20260924 02:57:49.812673 4587 tablet_service.cc:1467] Processing CreateTablet for tablet ec160828659c41a0b2a26a93050ef767 (DEFAULT_TABLE table=test-table [id=7a45790b69bc465ca23c930fea58b3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:49.812788 4934 tablet_bootstrap.cc:654] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:49.812861 4933 tablet_bootstrap.cc:654] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:49.812973 4587 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ec160828659c41a0b2a26a93050ef767. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:49.813283 4934 log.cc:824] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:49.813494 4933 log.cc:824] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:49.813956 4934 tablet_bootstrap.cc:492] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82: No bootstrap required, opened a new log
I20260924 02:57:49.814014 4934 ts_tablet_manager.cc:1397] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:49.814175 4933 tablet_bootstrap.cc:492] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c: No bootstrap required, opened a new log
I20260924 02:57:49.814244 4933 ts_tablet_manager.cc:1397] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:49.815403 4934 raft_consensus.cc:348] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da262eb236f4ac7b55d2b0d7fc11d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "b96e743a5ef048f3ad163e3796dbc586" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37517 } } peers { permanent_uuid: "72fd550323c045a48f7e0b83b47a778c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38297 } }
I20260924 02:57:49.815531 4934 raft_consensus.cc:374] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:49.815562 4934 raft_consensus.cc:729] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5da262eb236f4ac7b55d2b0d7fc11d82, State: Initialized, Role: FOLLOWER
I20260924 02:57:49.815673 4934 consensus_queue.cc:260] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da262eb236f4ac7b55d2b0d7fc11d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "b96e743a5ef048f3ad163e3796dbc586" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37517 } } peers { permanent_uuid: "72fd550323c045a48f7e0b83b47a778c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38297 } }
I20260924 02:57:49.815949 4934 ts_tablet_manager.cc:1428] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 02:57:49.816149 4786 heartbeater.cc:503] Master 127.0.98.62:33157 was elected leader, sending a full tablet report...
I20260924 02:57:49.816128 4933 raft_consensus.cc:348] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da262eb236f4ac7b55d2b0d7fc11d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "b96e743a5ef048f3ad163e3796dbc586" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37517 } } peers { permanent_uuid: "72fd550323c045a48f7e0b83b47a778c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38297 } }
I20260924 02:57:49.816269 4933 raft_consensus.cc:374] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:49.816306 4933 raft_consensus.cc:729] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72fd550323c045a48f7e0b83b47a778c, State: Initialized, Role: FOLLOWER
I20260924 02:57:49.816293 4937 tablet_bootstrap.cc:492] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586: Bootstrap starting.
I20260924 02:57:49.816406 4933 consensus_queue.cc:260] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da262eb236f4ac7b55d2b0d7fc11d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "b96e743a5ef048f3ad163e3796dbc586" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37517 } } peers { permanent_uuid: "72fd550323c045a48f7e0b83b47a778c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38297 } }
I20260924 02:57:49.816665 4933 ts_tablet_manager.cc:1428] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 02:57:49.816673 4919 heartbeater.cc:503] Master 127.0.98.62:33157 was elected leader, sending a full tablet report...
I20260924 02:57:49.816851 4937 tablet_bootstrap.cc:654] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:49.817373 4937 log.cc:824] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:49.817945 4937 tablet_bootstrap.cc:492] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586: No bootstrap required, opened a new log
I20260924 02:57:49.817994 4937 ts_tablet_manager.cc:1397] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:49.819352 4937 raft_consensus.cc:348] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da262eb236f4ac7b55d2b0d7fc11d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "b96e743a5ef048f3ad163e3796dbc586" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37517 } } peers { permanent_uuid: "72fd550323c045a48f7e0b83b47a778c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38297 } }
I20260924 02:57:49.819500 4937 raft_consensus.cc:374] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:49.819543 4937 raft_consensus.cc:729] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b96e743a5ef048f3ad163e3796dbc586, State: Initialized, Role: FOLLOWER
I20260924 02:57:49.819653 4937 consensus_queue.cc:260] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da262eb236f4ac7b55d2b0d7fc11d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "b96e743a5ef048f3ad163e3796dbc586" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37517 } } peers { permanent_uuid: "72fd550323c045a48f7e0b83b47a778c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38297 } }
I20260924 02:57:49.819880 4937 ts_tablet_manager.cc:1428] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 02:57:49.819906 4653 heartbeater.cc:503] Master 127.0.98.62:33157 was elected leader, sending a full tablet report...
W20260924 02:57:49.890661 4787 tablet.cc:2369] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 02:57:49.993213 4941 raft_consensus.cc:482] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:57:49.993335 4941 raft_consensus.cc:504] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da262eb236f4ac7b55d2b0d7fc11d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "b96e743a5ef048f3ad163e3796dbc586" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37517 } } peers { permanent_uuid: "72fd550323c045a48f7e0b83b47a778c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38297 } }
I20260924 02:57:49.993624 4941 leader_election.cc:288] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5da262eb236f4ac7b55d2b0d7fc11d82 (127.0.98.2:33251), 72fd550323c045a48f7e0b83b47a778c (127.0.98.3:38297)
I20260924 02:57:49.996263 4929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.993855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59591 (local address 127.0.98.2:33251)
0924 02:57:49.993925 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:49.993927 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:49.993937 (+ 10us) server_negotiation.cc:407] Connection header received
0924 02:57:49.993967 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:49.993968 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:49.993991 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:49.994056 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:49.994219 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.994729 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.995634 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.995980 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.996038 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.996048 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:49.996169 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 02:57:49.996201 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 02:57:49.996287 4932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.993972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:38807 (local address 127.0.98.3:38297)
0924 02:57:49.994035 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:49.994037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:49.994045 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:57:49.994068 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:49.994069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:49.994080 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:49.994124 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:49.994262 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.994753 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.995642 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.996000 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.996058 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.996068 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:49.996189 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 02:57:49.996223 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 02:57:49.996299 4655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.993782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:33251 (local address 127.0.98.1:59591)
0924 02:57:49.993834 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:49.993846 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:49.993888 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:49.994078 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:49.994079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:49.994081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:49.994135 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:49.994137 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.994813 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.994814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:49.995555 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:49.995557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.996045 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.996046 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:49.996114 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.996117 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:49.996159 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 02:57:49.996195 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 02:57:49.996529 4942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:49.993882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38297 (local address 127.0.98.1:38807)
0924 02:57:49.993968 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:49.993975 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:49.994011 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:49.994139 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:49.994140 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:49.994141 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:49.994206 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:49.994208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.994828 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.994829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:49.995564 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:49.995566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:49.996067 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:49.996068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:49.996132 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:49.996136 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:57:49.996179 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 02:57:49.996216 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":13,"thread_start_us":62,"threads_started":1}
I20260924 02:57:49.996663 4740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec160828659c41a0b2a26a93050ef767" candidate_uuid: "b96e743a5ef048f3ad163e3796dbc586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da262eb236f4ac7b55d2b0d7fc11d82" is_pre_election: true
I20260924 02:57:49.996819 4740 raft_consensus.cc:2454] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b96e743a5ef048f3ad163e3796dbc586 in term 0.
I20260924 02:57:49.996901 4873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec160828659c41a0b2a26a93050ef767" candidate_uuid: "b96e743a5ef048f3ad163e3796dbc586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72fd550323c045a48f7e0b83b47a778c" is_pre_election: true
I20260924 02:57:49.997030 4543 leader_election.cc:302] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5da262eb236f4ac7b55d2b0d7fc11d82, b96e743a5ef048f3ad163e3796dbc586; no voters:
I20260924 02:57:49.997048 4873 raft_consensus.cc:2454] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b96e743a5ef048f3ad163e3796dbc586 in term 0.
I20260924 02:57:49.997184 4941 raft_consensus.cc:2781] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:57:49.997249 4941 raft_consensus.cc:482] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:57:49.997272 4941 raft_consensus.cc:3037] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:49.997972 4941 raft_consensus.cc:504] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da262eb236f4ac7b55d2b0d7fc11d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "b96e743a5ef048f3ad163e3796dbc586" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37517 } } peers { permanent_uuid: "72fd550323c045a48f7e0b83b47a778c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38297 } }
I20260924 02:57:49.998095 4941 leader_election.cc:288] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [CANDIDATE]: Term 1 election: Requested vote from peers 5da262eb236f4ac7b55d2b0d7fc11d82 (127.0.98.2:33251), 72fd550323c045a48f7e0b83b47a778c (127.0.98.3:38297)
I20260924 02:57:49.998232 4740 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec160828659c41a0b2a26a93050ef767" candidate_uuid: "b96e743a5ef048f3ad163e3796dbc586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5da262eb236f4ac7b55d2b0d7fc11d82"
I20260924 02:57:49.998252 4873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ec160828659c41a0b2a26a93050ef767" candidate_uuid: "b96e743a5ef048f3ad163e3796dbc586" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72fd550323c045a48f7e0b83b47a778c"
I20260924 02:57:49.998310 4873 raft_consensus.cc:3037] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:49.998304 4740 raft_consensus.cc:3037] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:49.998993 4873 raft_consensus.cc:2454] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b96e743a5ef048f3ad163e3796dbc586 in term 1.
I20260924 02:57:49.999037 4740 raft_consensus.cc:2454] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b96e743a5ef048f3ad163e3796dbc586 in term 1.
I20260924 02:57:49.999137 4543 leader_election.cc:302] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72fd550323c045a48f7e0b83b47a778c, b96e743a5ef048f3ad163e3796dbc586; no voters:
I20260924 02:57:49.999214 4941 raft_consensus.cc:2781] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:49.999351 4941 raft_consensus.cc:686] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [term 1 LEADER]: Becoming Leader. State: Replica: b96e743a5ef048f3ad163e3796dbc586, State: Running, Role: LEADER
I20260924 02:57:49.999431 4941 consensus_queue.cc:237] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da262eb236f4ac7b55d2b0d7fc11d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } } peers { permanent_uuid: "b96e743a5ef048f3ad163e3796dbc586" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37517 } } peers { permanent_uuid: "72fd550323c045a48f7e0b83b47a778c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38297 } }
I20260924 02:57:50.000221 4462 catalog_manager.cc:5495] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 reported cstate change: term changed from 0 to 1, leader changed from <none> to b96e743a5ef048f3ad163e3796dbc586 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "b96e743a5ef048f3ad163e3796dbc586" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5da262eb236f4ac7b55d2b0d7fc11d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b96e743a5ef048f3ad163e3796dbc586" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37517 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72fd550323c045a48f7e0b83b47a778c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38297 } health_report { overall_health: UNKNOWN } } }
W20260924 02:57:50.001394 4654 tablet.cc:2369] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 02:57:50.026707 4920 tablet.cc:2369] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 02:57:50.036108 4930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.033861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35276 (local address 127.0.98.1:37517)
0924 02:57:50.033924 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:50.033926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:50.033935 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:57:50.033972 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:50.033973 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:50.033997 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:50.034056 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:50.034185 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:50.034663 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:50.035381 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:50.035572 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:50.035625 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:50.035770 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:50.035875 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:50.035945 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:50.035982 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 02:57:50.036001 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 02:57:50.037720 4740 raft_consensus.cc:1261] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82 [term 1 FOLLOWER]: Refusing update from remote peer b96e743a5ef048f3ad163e3796dbc586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:50.037720 4873 raft_consensus.cc:1261] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c [term 1 FOLLOWER]: Refusing update from remote peer b96e743a5ef048f3ad163e3796dbc586: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:50.038038 4941 consensus_queue.cc:1035] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72fd550323c045a48f7e0b83b47a778c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38297 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:50.038184 4943 consensus_queue.cc:1035] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5da262eb236f4ac7b55d2b0d7fc11d82" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 33251 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:50.040434 4948 mvcc.cc:205] Tried to move back new op lower bound from 7332735672471965696 to 7332735672318361600. Current Snapshot: MvccSnapshot[applied={T|T < 7332735672471965696}]
I20260924 02:57:50.042608 4463 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:50.042685 4463 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46020:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:50.043340 4463 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e272c8d5c83640b9a8002fc448f55e29: Sending DeleteTablet for 3 replicas of tablet ec160828659c41a0b2a26a93050ef767
I20260924 02:57:50.043663 4853 tablet_service.cc:1514] Processing DeleteTablet for tablet ec160828659c41a0b2a26a93050ef767 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36168
I20260924 02:57:50.043679 4720 tablet_service.cc:1514] Processing DeleteTablet for tablet ec160828659c41a0b2a26a93050ef767 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50300
I20260924 02:57:50.043674 4587 tablet_service.cc:1514] Processing DeleteTablet for tablet ec160828659c41a0b2a26a93050ef767 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35260
I20260924 02:57:50.043893 4958 tablet_replica.cc:330] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82: stopping tablet replica
I20260924 02:57:50.043990 4958 raft_consensus.cc:2229] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:50.043993 4957 tablet_replica.cc:330] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586: stopping tablet replica
I20260924 02:57:50.044081 4957 raft_consensus.cc:2229] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [term 1 LEADER]: Raft consensus shutting down.
I20260924 02:57:50.044087 4958 raft_consensus.cc:2258] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:50.044167 4957 raft_consensus.cc:2258] T ec160828659c41a0b2a26a93050ef767 P b96e743a5ef048f3ad163e3796dbc586 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:50.044083 4956 tablet_replica.cc:330] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c: stopping tablet replica
I20260924 02:57:50.044159 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 4525
I20260924 02:57:50.044294 4956 raft_consensus.cc:2229] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:50.044430 4956 raft_consensus.cc:2258] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:50.045013 4956 ts_tablet_manager.cc:1905] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:50.046348 4958 ts_tablet_manager.cc:1905] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:50.046620 4956 ts_tablet_manager.cc:1918] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:57:50.046680 4956 log.cc:1196] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/ec160828659c41a0b2a26a93050ef767
I20260924 02:57:50.046919 4956 ts_tablet_manager.cc:1939] T ec160828659c41a0b2a26a93050ef767 P 72fd550323c045a48f7e0b83b47a778c: Deleting consensus metadata
I20260924 02:57:50.047303 4450 catalog_manager.cc:4841] TS 72fd550323c045a48f7e0b83b47a778c (127.0.98.3:38297): tablet ec160828659c41a0b2a26a93050ef767 (table test-table [id=7a45790b69bc465ca23c930fea58b3b3]) successfully deleted
I20260924 02:57:50.047834 4958 ts_tablet_manager.cc:1918] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:57:50.047902 4958 log.cc:1196] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/ec160828659c41a0b2a26a93050ef767
I20260924 02:57:50.048141 4958 ts_tablet_manager.cc:1939] T ec160828659c41a0b2a26a93050ef767 P 5da262eb236f4ac7b55d2b0d7fc11d82: Deleting consensus metadata
I20260924 02:57:50.048501 4450 catalog_manager.cc:4841] TS 5da262eb236f4ac7b55d2b0d7fc11d82 (127.0.98.2:33251): tablet ec160828659c41a0b2a26a93050ef767 (table test-table [id=7a45790b69bc465ca23c930fea58b3b3]) successfully deleted
W20260924 02:57:50.049553 4448 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37517 (error 108)
I20260924 02:57:50.049747 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 4658
I20260924 02:57:50.050235 4931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.049836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37517 (local address 127.0.0.1:35288)
0924 02:57:50.050164 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:50.050201 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37517: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309}
W20260924 02:57:50.050482 4448 catalog_manager.cc:4584] TS b96e743a5ef048f3ad163e3796dbc586 (127.0.98.1:37517): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ec160828659c41a0b2a26a93050ef767: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37517: connect: Connection refused (error 111)
I20260924 02:57:50.053776 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 4791
I20260924 02:57:50.057477 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 4432
2026-09-24T02:57:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1916 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-24T02:57:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:57:50Z Disabled control of system clock
I20260924 02:57:50.082211 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43561
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:43561
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 02:57:50.158598 4963 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:50.158790 4963 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:50.158808 4963 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:50.159889 4963 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:50.159925 4963 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:50.159936 4963 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:50.159947 4963 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:50.159958 4963 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:57:50.161230 4963 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:43561
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:43561
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4963
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:50.161435 4963 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:50.161677 4963 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:50.166218 4971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:50.166217 4969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:50.166282 4968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:50.166342 4963 server_base.cc:1034] running on GCE node
I20260924 02:57:50.166724 4963 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:50.166998 4963 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:50.168147 4963 hybrid_clock.cc:648] HybridClock initialized: now 1790218670168126 us; error 35 us; skew 500 ppm
I20260924 02:57:50.169411 4963 webserver.cc:460] Webserver started at http://127.0.98.62:34193/ using document root <none> and password file <none>
I20260924 02:57:50.169626 4963 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:50.169672 4963 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:50.169827 4963 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:50.170957 4963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "fae12a77b43f44ee8689a0bef5183455"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "59baf2cde6fee80ea8e1c2b3e28a2d3e"
server_key_iv: "3d3979542ef22e423872bef168ecc84f"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.171279 4963 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "fae12a77b43f44ee8689a0bef5183455"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "59baf2cde6fee80ea8e1c2b3e28a2d3e"
server_key_iv: "3d3979542ef22e423872bef168ecc84f"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.172449 4963 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:57:50.173069 4977 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:50.173208 4963 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:50.173285 4963 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "fae12a77b43f44ee8689a0bef5183455"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "59baf2cde6fee80ea8e1c2b3e28a2d3e"
server_key_iv: "3d3979542ef22e423872bef168ecc84f"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.173345 4963 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:50.176601 4963 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:50.176843 4963 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:50.176956 4963 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:50.180342 4963 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:43561
I20260924 02:57:50.180371 5029 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:43561 every 8 connection(s)
I20260924 02:57:50.180636 4963 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:57:50.181193 5030 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:50.183748 5030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455: Bootstrap starting.
I20260924 02:57:50.184233 5030 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:50.184448 5030 log.cc:824] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:50.185058 5030 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455: No bootstrap required, opened a new log
I20260924 02:57:50.185721 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 4963
I20260924 02:57:50.185819 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:57:50.185968 392 external_mini_cluster.cc:1428] Setting key 7390d8e7ccd4c22482cbe899c8a00714
I20260924 02:57:50.186873 5030 raft_consensus.cc:348] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fae12a77b43f44ee8689a0bef5183455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } }
I20260924 02:57:50.187012 5030 raft_consensus.cc:374] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:50.187047 5030 raft_consensus.cc:729] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fae12a77b43f44ee8689a0bef5183455, State: Initialized, Role: FOLLOWER
I20260924 02:57:50.187168 5030 consensus_queue.cc:260] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fae12a77b43f44ee8689a0bef5183455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } }
I20260924 02:57:50.187261 5030 raft_consensus.cc:388] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:57:50.187294 5030 raft_consensus.cc:482] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:57:50.187340 5030 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:50.187929 5030 raft_consensus.cc:504] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fae12a77b43f44ee8689a0bef5183455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } }
I20260924 02:57:50.187997 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.186439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.0.98.62:43561)
0924 02:57:50.186678 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:50.186685 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:50.186707 (+ 22us) server_negotiation.cc:407] Connection header received
0924 02:57:50.186836 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:50.186841 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:50.186880 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:50.186982 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:50.187607 (+ 625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:50.187610 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:50.187610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:57:50.187614 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:50.187648 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:57:50.187658 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:50.187706 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:50.187779 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 02:57:50.187826 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":135,"threads_started":1}
I20260924 02:57:50.188019 5030 leader_election.cc:302] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fae12a77b43f44ee8689a0bef5183455; no voters:
I20260924 02:57:50.188161 5030 leader_election.cc:288] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:57:50.188391 5030 sys_catalog.cc:564] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:50.188431 5035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:50.188529 5035 raft_consensus.cc:686] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [term 1 LEADER]: Becoming Leader. State: Replica: fae12a77b43f44ee8689a0bef5183455, State: Running, Role: LEADER
I20260924 02:57:50.188608 5035 consensus_queue.cc:237] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fae12a77b43f44ee8689a0bef5183455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } }
I20260924 02:57:50.188922 5035 sys_catalog.cc:455] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fae12a77b43f44ee8689a0bef5183455. Latest consensus state: current_term: 1 leader_uuid: "fae12a77b43f44ee8689a0bef5183455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fae12a77b43f44ee8689a0bef5183455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } } }
I20260924 02:57:50.188997 5035 sys_catalog.cc:458] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:50.188952 5036 sys_catalog.cc:455] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fae12a77b43f44ee8689a0bef5183455" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fae12a77b43f44ee8689a0bef5183455" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 43561 } } }
I20260924 02:57:50.189096 5036 sys_catalog.cc:458] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:50.189179 5044 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:50.189672 5044 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:57:50.191000 5044 catalog_manager.cc:1347] Generated new cluster ID: 987508df2521454ebe675c531bb2fc19
I20260924 02:57:50.191052 5044 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:57:50.196327 5044 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:57:50.196461 5044 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:50.199357 5044 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455: Generated new TSK 0
I20260924 02:57:50.199501 5044 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 02:57:50.203207 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--builtin_ntp_servers=127.0.98.20:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260924 02:57:50.280926 5054 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:50.281116 5054 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:50.281136 5054 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:50.282250 5054 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:50.282286 5054 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:50.282310 5054 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:57:50.283612 5054 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5054
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:50.283825 5054 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:50.284096 5054 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:50.288729 5059 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:50.288738 5060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:50.288975 5054 server_base.cc:1034] running on GCE node
W20260924 02:57:50.289086 5062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:50.289299 5054 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:50.289520 5054 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:50.290668 5054 hybrid_clock.cc:648] HybridClock initialized: now 1790218670290654 us; error 32 us; skew 500 ppm
I20260924 02:57:50.291816 5054 webserver.cc:460] Webserver started at http://127.0.98.1:37649/ using document root <none> and password file <none>
I20260924 02:57:50.292025 5054 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:50.292069 5054 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:50.292196 5054 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:50.293195 5054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "26127d0c50704f9f89d6aff93f37db04"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "d73f85c5744bcd575d1bc8accd8c3c88"
server_key_iv: "dda9c58f0b01e2002c56747f5a9e4710"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.293498 5054 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "26127d0c50704f9f89d6aff93f37db04"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "d73f85c5744bcd575d1bc8accd8c3c88"
server_key_iv: "dda9c58f0b01e2002c56747f5a9e4710"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.294699 5054 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 02:57:50.295372 5068 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:50.295507 5054 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:50.295574 5054 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "26127d0c50704f9f89d6aff93f37db04"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "d73f85c5744bcd575d1bc8accd8c3c88"
server_key_iv: "dda9c58f0b01e2002c56747f5a9e4710"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.295635 5054 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:50.298844 5054 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:50.299079 5054 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:50.299186 5054 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:50.299367 5054 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:50.299664 5054 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:50.299700 5054 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:50.299718 5054 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:50.299741 5054 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:50.305974 5054 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39181
I20260924 02:57:50.306035 5181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39181 every 8 connection(s)
I20260924 02:57:50.306339 5054 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:57:50.307281 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 5054
I20260924 02:57:50.307401 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:57:50.307583 392 external_mini_cluster.cc:1428] Setting key fd15afef5e61e77d7731e286e7a616a2
I20260924 02:57:50.308457 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--builtin_ntp_servers=127.0.98.20:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 02:57:50.309625 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.307003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:53229 (local address 127.0.98.62:43561)
0924 02:57:50.307066 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:50.307068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:50.307319 (+ 251us) server_negotiation.cc:407] Connection header received
0924 02:57:50.307861 (+ 542us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:50.307862 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:50.307869 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:50.307916 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:50.309367 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:50.309369 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:50.309369 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:57:50.309374 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:50.309392 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:57:50.309396 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:50.309459 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:50.309536 (+ 77us) server_negotiation.cc:299] Negotiation successful
0924 02:57:50.309551 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 02:57:50.309827 5184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.306987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43561 (local address 127.0.98.1:53229)
0924 02:57:50.307168 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:50.307218 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:50.307763 (+ 545us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:50.308230 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:50.308235 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:50.308244 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:50.309222 (+ 978us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 02:57:50.309223 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:50.309251 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 02:57:50.309278 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:50.309465 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:50.309470 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:50.309473 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:50.309546 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 02:57:50.309610 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260924 02:57:50.310231 5182 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43561
I20260924 02:57:50.310321 5182 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:50.310453 5182 heartbeater.cc:511] Master 127.0.98.62:43561 requested a full tablet report, sending...
I20260924 02:57:50.310921 4994 ts_manager.cc:194] Registered new tserver with Master: 26127d0c50704f9f89d6aff93f37db04 (127.0.98.1:39181)
W20260924 02:57:50.387149 5185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:50.387352 5185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:50.387372 5185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:50.388554 5185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:50.388593 5185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:50.388617 5185 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:57:50.389994 5185 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5185
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:50.390216 5185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:50.390492 5185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:50.395174 5193 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:50.395157 5190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:50.395167 5191 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:50.395190 5185 server_base.cc:1034] running on GCE node
I20260924 02:57:50.395543 5185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:50.395772 5185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:50.396903 5185 hybrid_clock.cc:648] HybridClock initialized: now 1790218670396896 us; error 26 us; skew 500 ppm
I20260924 02:57:50.398000 5185 webserver.cc:460] Webserver started at http://127.0.98.2:44179/ using document root <none> and password file <none>
I20260924 02:57:50.398213 5185 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:50.398293 5185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:50.398422 5185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:50.399451 5185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "d2db491ddc2540889f2fb9a49b2d8eca"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "572dd9c0b2c62ac3a951ad04be1e7a89"
server_key_iv: "7fb453731f10e909d6154bee1df85e56"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.399724 5185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "d2db491ddc2540889f2fb9a49b2d8eca"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "572dd9c0b2c62ac3a951ad04be1e7a89"
server_key_iv: "7fb453731f10e909d6154bee1df85e56"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.400820 5185 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 02:57:50.401428 5199 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:50.401571 5185 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:50.401648 5185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "d2db491ddc2540889f2fb9a49b2d8eca"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "572dd9c0b2c62ac3a951ad04be1e7a89"
server_key_iv: "7fb453731f10e909d6154bee1df85e56"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.401707 5185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:50.404850 5185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:50.405078 5185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:50.405190 5185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:50.405378 5185 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:50.405643 5185 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:50.405675 5185 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:50.405700 5185 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:50.405725 5185 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:50.411293 5185 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41239
I20260924 02:57:50.411330 5312 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41239 every 8 connection(s)
I20260924 02:57:50.411628 5185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 02:57:50.412787 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 5185
I20260924 02:57:50.412880 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:57:50.413048 392 external_mini_cluster.cc:1428] Setting key 7d07f3ea98ec00e9837b872e943450a3
I20260924 02:57:50.413923 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--builtin_ntp_servers=127.0.98.20:43153
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260924 02:57:50.414547 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.412246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56559 (local address 127.0.98.62:43561)
0924 02:57:50.412307 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:50.412309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:50.412523 (+ 214us) server_negotiation.cc:407] Connection header received
0924 02:57:50.413034 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:50.413035 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:50.413042 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:50.413109 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:50.414294 (+ 1185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:50.414296 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:50.414296 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:57:50.414315 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:50.414334 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:57:50.414339 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:50.414399 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:50.414451 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 02:57:50.414464 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 02:57:50.414747 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.412251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43561 (local address 127.0.98.2:56559)
0924 02:57:50.412413 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:50.412446 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:50.412935 (+ 489us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:50.413185 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:50.413189 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:50.413199 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:50.414154 (+ 955us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 02:57:50.414155 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:50.414183 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 02:57:50.414212 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:50.414401 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:50.414403 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:50.414405 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 02:57:50.414455 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 02:57:50.414503 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":76,"threads_started":1}
I20260924 02:57:50.415261 5313 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43561
I20260924 02:57:50.415349 5313 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:50.415488 5313 heartbeater.cc:511] Master 127.0.98.62:43561 requested a full tablet report, sending...
I20260924 02:57:50.415812 4994 ts_manager.cc:194] Registered new tserver with Master: d2db491ddc2540889f2fb9a49b2d8eca (127.0.98.2:41239)
W20260924 02:57:50.492988 5316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:50.493185 5316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:50.493206 5316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:50.494324 5316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:50.494390 5316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:50.494416 5316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:57:50.495744 5316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43153
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:43561
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:50.495978 5316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:50.496244 5316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:50.500880 5321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:50.500905 5324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:50.501016 5316 server_base.cc:1034] running on GCE node
W20260924 02:57:50.500903 5322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:50.501318 5316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:50.501550 5316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:50.502679 5316 hybrid_clock.cc:648] HybridClock initialized: now 1790218670502667 us; error 27 us; skew 500 ppm
I20260924 02:57:50.503773 5316 webserver.cc:460] Webserver started at http://127.0.98.3:45009/ using document root <none> and password file <none>
I20260924 02:57:50.504025 5316 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:50.504073 5316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:50.504204 5316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:50.505257 5316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "988e43b7b560411e84264b10c37605d0"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "c39aad437e2e35339b198230f6f6877f"
server_key_iv: "aafd727f056e39c7255556dd855ba267"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.505584 5316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "988e43b7b560411e84264b10c37605d0"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "c39aad437e2e35339b198230f6f6877f"
server_key_iv: "aafd727f056e39c7255556dd855ba267"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.506847 5316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 02:57:50.507493 5330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:50.507644 5316 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:50.507723 5316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "988e43b7b560411e84264b10c37605d0"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "c39aad437e2e35339b198230f6f6877f"
server_key_iv: "aafd727f056e39c7255556dd855ba267"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.507781 5316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:50.511444 5316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:50.511677 5316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:50.511787 5316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:50.511966 5316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:50.512228 5316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:50.512259 5316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:50.512284 5316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:50.512308 5316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:50.517422 5316 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42709
I20260924 02:57:50.517455 5443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42709 every 8 connection(s)
I20260924 02:57:50.517740 5316 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 02:57:50.518581 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 5316
I20260924 02:57:50.518685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:57:50.518864 392 external_mini_cluster.cc:1428] Setting key e9b0876954041f19b133a81adcdcad55
I20260924 02:57:50.520581 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.518396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:47947 (local address 127.0.98.62:43561)
0924 02:57:50.518436 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:50.518439 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:50.518709 (+ 270us) server_negotiation.cc:407] Connection header received
0924 02:57:50.519179 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:50.519180 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:50.519186 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:50.519229 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:50.520350 (+ 1121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:50.520351 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:50.520352 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:57:50.520356 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:50.520372 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:57:50.520376 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:50.520424 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:50.520499 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 02:57:50.520514 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 02:57:50.520756 5446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.518405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:43561 (local address 127.0.98.3:47947)
0924 02:57:50.518597 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:50.518631 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:50.519095 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:50.519297 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:50.519301 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:50.519311 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:50.520215 (+ 904us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 02:57:50.520215 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:50.520242 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 02:57:50.520269 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:50.520430 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:50.520435 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:50.520438 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:50.520499 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 02:57:50.520559 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260924 02:57:50.521211 5444 heartbeater.cc:347] Connected to a master server at 127.0.98.62:43561
I20260924 02:57:50.521284 5444 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:50.521392 5444 heartbeater.cc:511] Master 127.0.98.62:43561 requested a full tablet report, sending...
I20260924 02:57:50.521660 4994 ts_manager.cc:194] Registered new tserver with Master: 988e43b7b560411e84264b10c37605d0 (127.0.98.3:42709)
I20260924 02:57:50.530637 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 02:57:50.532820 5034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.531948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51510 (local address 127.0.98.62:43561)
0924 02:57:50.531995 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:50.531997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:50.532094 (+ 97us) server_negotiation.cc:407] Connection header received
0924 02:57:50.532140 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:50.532141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:50.532145 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:50.532194 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:50.532666 (+ 472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:50.532667 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:50.532667 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:57:50.532671 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:50.532686 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:57:50.532689 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:50.532727 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:50.532756 (+ 29us) server_negotiation.cc:299] Negotiation successful
0924 02:57:50.532767 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 02:57:50.534250 4994 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51510:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 02:57:50.535742 4994 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:57:50.539391 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.537719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41239 (local address 127.0.0.1:43382)
0924 02:57:50.537880 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:50.537921 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:50.537977 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:50.538473 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:50.538474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:50.538476 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:50.538910 (+ 434us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 02:57:50.538910 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:50.538927 (+ 17us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 02:57:50.538950 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:50.539217 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:50.539222 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:50.539225 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:50.539301 (+ 76us) client_negotiation.cc:240] Negotiation successful
0924 02:57:50.539313 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260924 02:57:50.539404 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.537815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43382 (local address 127.0.98.2:41239)
0924 02:57:50.538044 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:50.538049 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:50.538060 (+ 11us) server_negotiation.cc:407] Connection header received
0924 02:57:50.538085 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:50.538087 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:50.538098 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:50.538175 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:50.539101 (+ 926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:50.539107 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:50.539108 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:57:50.539111 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:50.539142 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:57:50.539166 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:50.539220 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:50.539307 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 02:57:50.539321 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":195,"thread_start_us":115,"threads_started":1}
I20260924 02:57:50.539610 5453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.537719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42709 (local address 127.0.0.1:43730)
0924 02:57:50.538107 (+ 388us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:50.538116 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:50.538183 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:50.538360 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:50.538368 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:50.538374 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:50.539158 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 02:57:50.539158 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:50.539163 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 02:57:50.539172 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:50.539376 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:50.539376 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:50.539377 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 02:57:50.539427 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 02:57:50.539551 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":372,"mutex_wait_us":2,"thread_start_us":89,"threads_started":1}
I20260924 02:57:50.539746 5457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.538107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53410 (local address 127.0.98.1:39181)
0924 02:57:50.538304 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:50.538307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:50.538314 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:57:50.538334 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:50.538336 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:50.538345 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:50.538410 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:50.539089 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:50.539097 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:50.539097 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:57:50.539101 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:50.539135 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:57:50.539160 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:50.539220 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:50.539650 (+ 430us) server_negotiation.cc:299] Negotiation successful
0924 02:57:50.539667 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":92,"threads_started":1}
I20260924 02:57:50.539884 5456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.537917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39181 (local address 127.0.0.1:53410)
0924 02:57:50.538118 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:50.538128 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:50.538164 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:50.538704 (+ 540us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:50.538704 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:50.538707 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:50.539006 (+ 299us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 02:57:50.539006 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:50.539012 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 02:57:50.539021 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:50.539562 (+ 541us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:50.539562 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:50.539563 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 02:57:50.539604 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 02:57:50.539614 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":56,"threads_started":1}
I20260924 02:57:50.540094 5247 tablet_service.cc:1467] Processing CreateTablet for tablet 6facfbae916f4772aaf88d1fc5a35dc2 (DEFAULT_TABLE table=test-table [id=6b96728fe2e1459e8f94fa3e16b40132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:50.540426 5247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6facfbae916f4772aaf88d1fc5a35dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:50.540504 5116 tablet_service.cc:1467] Processing CreateTablet for tablet 6facfbae916f4772aaf88d1fc5a35dc2 (DEFAULT_TABLE table=test-table [id=6b96728fe2e1459e8f94fa3e16b40132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:50.540815 5116 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6facfbae916f4772aaf88d1fc5a35dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:50.542922 5458 tablet_bootstrap.cc:492] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca: Bootstrap starting.
I20260924 02:57:50.543190 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.537888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43730 (local address 127.0.98.3:42709)
0924 02:57:50.538054 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:50.538059 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:50.538164 (+ 105us) server_negotiation.cc:407] Connection header received
0924 02:57:50.538263 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:50.538270 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:50.538280 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:50.538352 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:50.539243 (+ 891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:50.539250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:50.539251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:57:50.539255 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:50.539287 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:57:50.539315 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:50.539369 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:50.539435 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 02:57:50.539449 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":71,"threads_started":1}
I20260924 02:57:50.543257 5459 tablet_bootstrap.cc:492] T 6facfbae916f4772aaf88d1fc5a35dc2 P 26127d0c50704f9f89d6aff93f37db04: Bootstrap starting.
I20260924 02:57:50.543687 5458 tablet_bootstrap.cc:654] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:50.543854 5459 tablet_bootstrap.cc:654] T 6facfbae916f4772aaf88d1fc5a35dc2 P 26127d0c50704f9f89d6aff93f37db04: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:50.543773 5378 tablet_service.cc:1467] Processing CreateTablet for tablet 6facfbae916f4772aaf88d1fc5a35dc2 (DEFAULT_TABLE table=test-table [id=6b96728fe2e1459e8f94fa3e16b40132]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:50.543967 5458 log.cc:824] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:50.544095 5378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6facfbae916f4772aaf88d1fc5a35dc2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:50.544155 5459 log.cc:824] T 6facfbae916f4772aaf88d1fc5a35dc2 P 26127d0c50704f9f89d6aff93f37db04: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:50.544625 5458 tablet_bootstrap.cc:492] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca: No bootstrap required, opened a new log
I20260924 02:57:50.544682 5458 ts_tablet_manager.cc:1397] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:50.544832 5459 tablet_bootstrap.cc:492] T 6facfbae916f4772aaf88d1fc5a35dc2 P 26127d0c50704f9f89d6aff93f37db04: No bootstrap required, opened a new log
I20260924 02:57:50.544899 5459 ts_tablet_manager.cc:1397] T 6facfbae916f4772aaf88d1fc5a35dc2 P 26127d0c50704f9f89d6aff93f37db04: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:50.546657 5458 raft_consensus.cc:348] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988e43b7b560411e84264b10c37605d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42709 } } peers { permanent_uuid: "d2db491ddc2540889f2fb9a49b2d8eca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41239 } } peers { permanent_uuid: "26127d0c50704f9f89d6aff93f37db04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39181 } }
I20260924 02:57:50.546780 5458 raft_consensus.cc:374] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:50.546809 5458 raft_consensus.cc:729] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d2db491ddc2540889f2fb9a49b2d8eca, State: Initialized, Role: FOLLOWER
I20260924 02:57:50.546916 5458 consensus_queue.cc:260] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988e43b7b560411e84264b10c37605d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42709 } } peers { permanent_uuid: "d2db491ddc2540889f2fb9a49b2d8eca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41239 } } peers { permanent_uuid: "26127d0c50704f9f89d6aff93f37db04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39181 } }
I20260924 02:57:50.547152 5458 ts_tablet_manager.cc:1428] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 02:57:50.547292 5313 heartbeater.cc:503] Master 127.0.98.62:43561 was elected leader, sending a full tablet report...
I20260924 02:57:50.547355 5462 tablet_bootstrap.cc:492] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0: Bootstrap starting.
I20260924 02:57:50.547348 5459 raft_consensus.cc:348] T 6facfbae916f4772aaf88d1fc5a35dc2 P 26127d0c50704f9f89d6aff93f37db04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988e43b7b560411e84264b10c37605d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42709 } } peers { permanent_uuid: "d2db491ddc2540889f2fb9a49b2d8eca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41239 } } peers { permanent_uuid: "26127d0c50704f9f89d6aff93f37db04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39181 } }
I20260924 02:57:50.547508 5459 raft_consensus.cc:374] T 6facfbae916f4772aaf88d1fc5a35dc2 P 26127d0c50704f9f89d6aff93f37db04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:50.547542 5459 raft_consensus.cc:729] T 6facfbae916f4772aaf88d1fc5a35dc2 P 26127d0c50704f9f89d6aff93f37db04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26127d0c50704f9f89d6aff93f37db04, State: Initialized, Role: FOLLOWER
I20260924 02:57:50.547676 5459 consensus_queue.cc:260] T 6facfbae916f4772aaf88d1fc5a35dc2 P 26127d0c50704f9f89d6aff93f37db04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988e43b7b560411e84264b10c37605d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42709 } } peers { permanent_uuid: "d2db491ddc2540889f2fb9a49b2d8eca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41239 } } peers { permanent_uuid: "26127d0c50704f9f89d6aff93f37db04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39181 } }
I20260924 02:57:50.547914 5459 ts_tablet_manager.cc:1428] T 6facfbae916f4772aaf88d1fc5a35dc2 P 26127d0c50704f9f89d6aff93f37db04: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 02:57:50.547971 5462 tablet_bootstrap.cc:654] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:50.548164 5182 heartbeater.cc:503] Master 127.0.98.62:43561 was elected leader, sending a full tablet report...
I20260924 02:57:50.548257 5462 log.cc:824] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:50.548877 5462 tablet_bootstrap.cc:492] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0: No bootstrap required, opened a new log
I20260924 02:57:50.548938 5462 ts_tablet_manager.cc:1397] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:50.550750 5462 raft_consensus.cc:348] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988e43b7b560411e84264b10c37605d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42709 } } peers { permanent_uuid: "d2db491ddc2540889f2fb9a49b2d8eca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41239 } } peers { permanent_uuid: "26127d0c50704f9f89d6aff93f37db04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39181 } }
I20260924 02:57:50.550864 5462 raft_consensus.cc:374] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:50.550894 5462 raft_consensus.cc:729] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 988e43b7b560411e84264b10c37605d0, State: Initialized, Role: FOLLOWER
I20260924 02:57:50.550989 5462 consensus_queue.cc:260] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988e43b7b560411e84264b10c37605d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42709 } } peers { permanent_uuid: "d2db491ddc2540889f2fb9a49b2d8eca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41239 } } peers { permanent_uuid: "26127d0c50704f9f89d6aff93f37db04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39181 } }
I20260924 02:57:50.551182 5462 ts_tablet_manager.cc:1428] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 02:57:50.551223 5444 heartbeater.cc:503] Master 127.0.98.62:43561 was elected leader, sending a full tablet report...
W20260924 02:57:50.556799 5183 tablet.cc:2369] T 6facfbae916f4772aaf88d1fc5a35dc2 P 26127d0c50704f9f89d6aff93f37db04: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 02:57:50.610734 5463 raft_consensus.cc:482] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:57:50.610848 5463 raft_consensus.cc:504] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988e43b7b560411e84264b10c37605d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42709 } } peers { permanent_uuid: "d2db491ddc2540889f2fb9a49b2d8eca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41239 } } peers { permanent_uuid: "26127d0c50704f9f89d6aff93f37db04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39181 } }
I20260924 02:57:50.611213 5463 leader_election.cc:288] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 988e43b7b560411e84264b10c37605d0 (127.0.98.3:42709), 26127d0c50704f9f89d6aff93f37db04 (127.0.98.1:39181)
I20260924 02:57:50.612353 5455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.611455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37927 (local address 127.0.98.3:42709)
0924 02:57:50.611495 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:50.611497 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:50.611522 (+ 25us) server_negotiation.cc:407] Connection header received
0924 02:57:50.611544 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:50.611545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:50.611549 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:50.611601 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:50.612145 (+ 544us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:50.612147 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:50.612147 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:57:50.612151 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:50.612168 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:57:50.612174 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:50.612230 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:50.612270 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 02:57:50.612284 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 02:57:50.612360 5315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.611373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42709 (local address 127.0.98.2:37927)
0924 02:57:50.611415 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:50.611426 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:50.611460 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:50.611611 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:50.611612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:50.611615 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:50.612042 (+ 427us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 02:57:50.612043 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:50.612051 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 02:57:50.612065 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:50.612227 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:50.612228 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:50.612228 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 02:57:50.612260 (+ 32us) client_negotiation.cc:240] Negotiation successful
0924 02:57:50.612275 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 02:57:50.612538 5457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.611423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:56749 (local address 127.0.98.1:39181)
0924 02:57:50.611464 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:50.611467 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:50.611694 (+ 227us) server_negotiation.cc:407] Connection header received
0924 02:57:50.611741 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:50.611742 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:50.611748 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:50.611810 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:50.612299 (+ 489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:50.612300 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:50.612301 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:57:50.612305 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:50.612323 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:57:50.612327 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:50.612379 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:50.612462 (+ 83us) server_negotiation.cc:299] Negotiation successful
0924 02:57:50.612477 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 02:57:50.612709 5467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.611373 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39181 (local address 127.0.98.2:56749)
0924 02:57:50.611636 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:50.611645 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:50.611688 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:50.611829 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:50.611829 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:50.611832 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:50.612198 (+ 366us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 02:57:50.612198 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:50.612205 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 02:57:50.612215 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:50.612382 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:50.612383 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:50.612383 (+ 0us) client_negotiation.cc:769] Sending connection context
0924 02:57:50.612623 (+ 240us) client_negotiation.cc:240] Negotiation successful
0924 02:57:50.612637 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247,"mutex_wait_us":78,"thread_start_us":122,"threads_started":1}
I20260924 02:57:50.612658 5398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6facfbae916f4772aaf88d1fc5a35dc2" candidate_uuid: "d2db491ddc2540889f2fb9a49b2d8eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "988e43b7b560411e84264b10c37605d0" is_pre_election: true
I20260924 02:57:50.612804 5398 raft_consensus.cc:2454] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2db491ddc2540889f2fb9a49b2d8eca in term 0.
I20260924 02:57:50.613019 5202 leader_election.cc:302] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 988e43b7b560411e84264b10c37605d0, d2db491ddc2540889f2fb9a49b2d8eca; no voters:
I20260924 02:57:50.612998 5136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6facfbae916f4772aaf88d1fc5a35dc2" candidate_uuid: "d2db491ddc2540889f2fb9a49b2d8eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26127d0c50704f9f89d6aff93f37db04" is_pre_election: true
I20260924 02:57:50.613132 5136 raft_consensus.cc:2454] T 6facfbae916f4772aaf88d1fc5a35dc2 P 26127d0c50704f9f89d6aff93f37db04 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d2db491ddc2540889f2fb9a49b2d8eca in term 0.
I20260924 02:57:50.613142 5463 raft_consensus.cc:2781] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:57:50.613179 5463 raft_consensus.cc:482] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:57:50.613207 5463 raft_consensus.cc:3037] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:50.613894 5463 raft_consensus.cc:504] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988e43b7b560411e84264b10c37605d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42709 } } peers { permanent_uuid: "d2db491ddc2540889f2fb9a49b2d8eca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41239 } } peers { permanent_uuid: "26127d0c50704f9f89d6aff93f37db04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39181 } }
I20260924 02:57:50.614004 5463 leader_election.cc:288] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [CANDIDATE]: Term 1 election: Requested vote from peers 988e43b7b560411e84264b10c37605d0 (127.0.98.3:42709), 26127d0c50704f9f89d6aff93f37db04 (127.0.98.1:39181)
I20260924 02:57:50.614143 5398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6facfbae916f4772aaf88d1fc5a35dc2" candidate_uuid: "d2db491ddc2540889f2fb9a49b2d8eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "988e43b7b560411e84264b10c37605d0"
I20260924 02:57:50.614154 5136 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6facfbae916f4772aaf88d1fc5a35dc2" candidate_uuid: "d2db491ddc2540889f2fb9a49b2d8eca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26127d0c50704f9f89d6aff93f37db04"
I20260924 02:57:50.614213 5398 raft_consensus.cc:3037] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:50.614220 5136 raft_consensus.cc:3037] T 6facfbae916f4772aaf88d1fc5a35dc2 P 26127d0c50704f9f89d6aff93f37db04 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:50.614928 5136 raft_consensus.cc:2454] T 6facfbae916f4772aaf88d1fc5a35dc2 P 26127d0c50704f9f89d6aff93f37db04 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2db491ddc2540889f2fb9a49b2d8eca in term 1.
I20260924 02:57:50.614948 5398 raft_consensus.cc:2454] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d2db491ddc2540889f2fb9a49b2d8eca in term 1.
I20260924 02:57:50.615090 5203 leader_election.cc:302] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26127d0c50704f9f89d6aff93f37db04, d2db491ddc2540889f2fb9a49b2d8eca; no voters:
I20260924 02:57:50.615159 5463 raft_consensus.cc:2781] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:50.615309 5463 raft_consensus.cc:686] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [term 1 LEADER]: Becoming Leader. State: Replica: d2db491ddc2540889f2fb9a49b2d8eca, State: Running, Role: LEADER
I20260924 02:57:50.615376 5463 consensus_queue.cc:237] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988e43b7b560411e84264b10c37605d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42709 } } peers { permanent_uuid: "d2db491ddc2540889f2fb9a49b2d8eca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41239 } } peers { permanent_uuid: "26127d0c50704f9f89d6aff93f37db04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39181 } }
I20260924 02:57:50.616101 4994 catalog_manager.cc:5495] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca reported cstate change: term changed from 0 to 1, leader changed from <none> to d2db491ddc2540889f2fb9a49b2d8eca (127.0.98.2). New cstate: current_term: 1 leader_uuid: "d2db491ddc2540889f2fb9a49b2d8eca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "988e43b7b560411e84264b10c37605d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d2db491ddc2540889f2fb9a49b2d8eca" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41239 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26127d0c50704f9f89d6aff93f37db04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39181 } health_report { overall_health: UNKNOWN } } }
I20260924 02:57:50.630832 5454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.629849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43394 (local address 127.0.98.2:41239)
0924 02:57:50.629903 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:50.629905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:50.629913 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:57:50.629996 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:50.629996 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:50.630007 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:50.630049 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:50.630582 (+ 533us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:50.630584 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:50.630584 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:57:50.630589 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:50.630604 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:57:50.630607 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:50.630665 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:50.630733 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 02:57:50.630746 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 02:57:50.632470 5398 raft_consensus.cc:1261] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0 [term 1 FOLLOWER]: Refusing update from remote peer d2db491ddc2540889f2fb9a49b2d8eca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:50.632470 5136 raft_consensus.cc:1261] T 6facfbae916f4772aaf88d1fc5a35dc2 P 26127d0c50704f9f89d6aff93f37db04 [term 1 FOLLOWER]: Refusing update from remote peer d2db491ddc2540889f2fb9a49b2d8eca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:50.632784 5468 consensus_queue.cc:1035] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [LEADER]: Connected to new peer: Peer: permanent_uuid: "26127d0c50704f9f89d6aff93f37db04" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:50.632899 5463 consensus_queue.cc:1035] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [LEADER]: Connected to new peer: Peer: permanent_uuid: "988e43b7b560411e84264b10c37605d0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:50.635449 5472 mvcc.cc:205] Tried to move back new op lower bound from 7332735674908147712 to 7332735674841042944. Current Snapshot: MvccSnapshot[applied={T|T < 7332735674908147712}]
I20260924 02:57:50.637213 4993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:50.637297 4993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51510:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:50.637956 4993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fae12a77b43f44ee8689a0bef5183455: Sending DeleteTablet for 3 replicas of tablet 6facfbae916f4772aaf88d1fc5a35dc2
I20260924 02:57:50.638252 5247 tablet_service.cc:1514] Processing DeleteTablet for tablet 6facfbae916f4772aaf88d1fc5a35dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:50 UTC) from {username='slave'} at 127.0.0.1:43382
I20260924 02:57:50.638252 5378 tablet_service.cc:1514] Processing DeleteTablet for tablet 6facfbae916f4772aaf88d1fc5a35dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:50 UTC) from {username='slave'} at 127.0.0.1:43730
I20260924 02:57:50.638442 5480 tablet_replica.cc:330] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca: stopping tablet replica
I20260924 02:57:50.638469 5116 tablet_service.cc:1514] Processing DeleteTablet for tablet 6facfbae916f4772aaf88d1fc5a35dc2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:50 UTC) from {username='slave'} at 127.0.0.1:53410
I20260924 02:57:50.638556 5480 raft_consensus.cc:2229] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [term 1 LEADER]: Raft consensus shutting down.
I20260924 02:57:50.638676 5481 tablet_replica.cc:330] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0: stopping tablet replica
I20260924 02:57:50.638677 5480 raft_consensus.cc:2258] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:50.638882 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 5054
I20260924 02:57:50.638896 5481 raft_consensus.cc:2229] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:50.638998 5481 raft_consensus.cc:2258] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:50.639060 5480 ts_tablet_manager.cc:1905] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:50.639684 5481 ts_tablet_manager.cc:1905] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:50.640776 5480 ts_tablet_manager.cc:1918] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:57:50.640832 5480 log.cc:1196] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/6facfbae916f4772aaf88d1fc5a35dc2
I20260924 02:57:50.641083 5480 ts_tablet_manager.cc:1939] T 6facfbae916f4772aaf88d1fc5a35dc2 P d2db491ddc2540889f2fb9a49b2d8eca: Deleting consensus metadata
I20260924 02:57:50.641160 5481 ts_tablet_manager.cc:1918] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:57:50.641209 5481 log.cc:1196] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/6facfbae916f4772aaf88d1fc5a35dc2
I20260924 02:57:50.641418 5481 ts_tablet_manager.cc:1939] T 6facfbae916f4772aaf88d1fc5a35dc2 P 988e43b7b560411e84264b10c37605d0: Deleting consensus metadata
I20260924 02:57:50.641455 4981 catalog_manager.cc:4841] TS d2db491ddc2540889f2fb9a49b2d8eca (127.0.98.2:41239): tablet 6facfbae916f4772aaf88d1fc5a35dc2 (table test-table [id=6b96728fe2e1459e8f94fa3e16b40132]) successfully deleted
I20260924 02:57:50.641775 4980 catalog_manager.cc:4841] TS 988e43b7b560411e84264b10c37605d0 (127.0.98.3:42709): tablet 6facfbae916f4772aaf88d1fc5a35dc2 (table test-table [id=6b96728fe2e1459e8f94fa3e16b40132]) successfully deleted
W20260924 02:57:50.645963 4981 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39181 (error 108)
I20260924 02:57:50.646171 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 5185
I20260924 02:57:50.646440 5452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.646339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39181 (local address 127.0.0.1:53418)
0924 02:57:50.646381 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:50.646413 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39181: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 02:57:50.646728 4981 catalog_manager.cc:4584] TS 26127d0c50704f9f89d6aff93f37db04 (127.0.98.1:39181): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6facfbae916f4772aaf88d1fc5a35dc2: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39181: connect: Connection refused (error 111)
I20260924 02:57:50.650755 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 5316
I20260924 02:57:50.654258 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 4963
2026-09-24T02:57:50Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (596 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260924 02:57:50.672212 392 mini_oidc.cc:152] Starting JWKS server
I20260924 02:57:50.673209 392 webserver.cc:460] Webserver started at https://127.0.0.1:43543/ using document root <none> and password file <none>
I20260924 02:57:50.673287 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 02:57:50.673412 392 webserver.cc:460] Webserver started at http://127.0.0.1:39579/ using document root <none> and password file <none>
2026-09-24T02:57:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:57:50Z Disabled control of system clock
I20260924 02:57:50.680249 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36969
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:37107
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36969
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43543/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/testchainca.cert with env {}
W20260924 02:57:50.757368 5489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:50.757560 5489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:50.757581 5489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:50.758751 5489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:50.758791 5489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:50.758802 5489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:50.758814 5489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:50.758824 5489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 02:57:50.758838 5489 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 02:57:50.758852 5489 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43543/jwks.json
I20260924 02:57:50.760347 5489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37107
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36969
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36969
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43543/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:50.760545 5489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:50.760705 5489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:50.765121 5495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:50.765233 5494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:50.765131 5497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:50.765163 5489 server_base.cc:1034] running on GCE node
I20260924 02:57:50.765537 5489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:50.765795 5489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:50.766932 5489 hybrid_clock.cc:648] HybridClock initialized: now 1790218670766911 us; error 31 us; skew 500 ppm
I20260924 02:57:50.768085 5489 webserver.cc:460] Webserver started at http://127.0.98.62:45695/ using document root <none> and password file <none>
I20260924 02:57:50.768301 5489 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:50.768357 5489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:50.768474 5489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:50.769469 5489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "92e8235c2aa349b3a5f0a506e8f802de"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "2f4f6c1a860f4a4898439d6e3d344085"
server_key_iv: "7d74c05cfea7447b5ca832aaf53d0773"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.769785 5489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "92e8235c2aa349b3a5f0a506e8f802de"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "2f4f6c1a860f4a4898439d6e3d344085"
server_key_iv: "7d74c05cfea7447b5ca832aaf53d0773"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.770988 5489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 02:57:50.771694 5503 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:50.771857 5489 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:50.771938 5489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "92e8235c2aa349b3a5f0a506e8f802de"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "2f4f6c1a860f4a4898439d6e3d344085"
server_key_iv: "7d74c05cfea7447b5ca832aaf53d0773"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.772002 5489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:50.784646 5489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:50.784914 5489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:50.785033 5489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:50.788554 5489 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36969
I20260924 02:57:50.788585 5555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36969 every 8 connection(s)
I20260924 02:57:50.788879 5489 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:57:50.789707 5556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:50.791862 5556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de: Bootstrap starting.
I20260924 02:57:50.792356 5556 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:50.792582 5556 log.cc:824] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:50.793123 5556 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de: No bootstrap required, opened a new log
I20260924 02:57:50.793749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 5489
I20260924 02:57:50.793828 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:57:50.793978 392 external_mini_cluster.cc:1428] Setting key 05654630ac256062b269b744171e6aaf
I20260924 02:57:50.794881 5556 raft_consensus.cc:348] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e8235c2aa349b3a5f0a506e8f802de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } }
I20260924 02:57:50.795015 5556 raft_consensus.cc:374] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:50.795047 5556 raft_consensus.cc:729] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92e8235c2aa349b3a5f0a506e8f802de, State: Initialized, Role: FOLLOWER
I20260924 02:57:50.795183 5556 consensus_queue.cc:260] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e8235c2aa349b3a5f0a506e8f802de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } }
I20260924 02:57:50.795265 5556 raft_consensus.cc:388] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:57:50.795295 5556 raft_consensus.cc:482] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:57:50.795342 5556 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:50.795807 5556 raft_consensus.cc:504] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e8235c2aa349b3a5f0a506e8f802de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } }
I20260924 02:57:50.795884 5556 leader_election.cc:302] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 92e8235c2aa349b3a5f0a506e8f802de; no voters:
I20260924 02:57:50.796015 5556 leader_election.cc:288] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:57:50.796067 5561 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:50.796231 5561 raft_consensus.cc:686] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [term 1 LEADER]: Becoming Leader. State: Replica: 92e8235c2aa349b3a5f0a506e8f802de, State: Running, Role: LEADER
I20260924 02:57:50.796244 5556 sys_catalog.cc:564] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:50.796320 5561 consensus_queue.cc:237] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e8235c2aa349b3a5f0a506e8f802de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } }
I20260924 02:57:50.796591 5561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [sys.catalog]: SysCatalogTable state changed. Reason: New leader 92e8235c2aa349b3a5f0a506e8f802de. Latest consensus state: current_term: 1 leader_uuid: "92e8235c2aa349b3a5f0a506e8f802de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e8235c2aa349b3a5f0a506e8f802de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } }
I20260924 02:57:50.796645 5561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:50.796784 5562 sys_catalog.cc:455] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "92e8235c2aa349b3a5f0a506e8f802de" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "92e8235c2aa349b3a5f0a506e8f802de" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36969 } } }
I20260924 02:57:50.796846 5562 sys_catalog.cc:458] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de [sys.catalog]: This master's current role is: LEADER
W20260924 02:57:50.797866 5577 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 02:57:50.797910 5577 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 02:57:50.797961 5567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:50.798117 5567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:57:50.799487 5567 catalog_manager.cc:1347] Generated new cluster ID: eacec62239f14f9aae236d97678efa97
I20260924 02:57:50.799533 5567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:57:50.799774 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.794418 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58130 (local address 127.0.98.62:36969)
0924 02:57:50.794609 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:50.794616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:50.794639 (+ 23us) server_negotiation.cc:407] Connection header received
0924 02:57:50.794856 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:50.794861 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:50.794914 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:50.795021 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:50.795231 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:50.796713 (+ 1482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:50.797757 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:50.798363 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:50.798482 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:50.799183 (+ 701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:50.799187 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:50.799188 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:57:50.799340 (+ 152us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:50.799370 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:57:50.799380 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:50.799440 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:50.799541 (+ 101us) server_negotiation.cc:299] Negotiation successful
0924 02:57:50.799615 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":60,"threads_started":1}
I20260924 02:57:50.803277 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 02:57:50.812780 5567 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:57:50.813438 5567 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:50.825080 5567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 92e8235c2aa349b3a5f0a506e8f802de: Generated new TSK 0
I20260924 02:57:50.825271 5567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 02:57:50.853335 5560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:50.830479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58138 (local address 127.0.98.62:36969)
0924 02:57:50.830546 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:50.830548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:50.830625 (+ 77us) server_negotiation.cc:407] Connection header received
0924 02:57:50.830711 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:50.830712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:50.830724 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:50.830777 (+ 53us) server_negotiation.cc:226] Negotiated authn=JWT
0924 02:57:50.830950 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:50.831438 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:50.832169 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:50.832364 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:50.832415 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:50.832595 (+ 180us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 02:57:50.853104 (+ 20509us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 02:57:50.853260 (+ 156us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58138: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 02:57:50.853405 5560 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58138: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260924 02:57:50.853681 5585 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:57:50.830388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36969 (local address 127.0.0.1:58138)
0924 02:57:50.830562 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:50.830573 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:50.830635 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:50.830826 (+ 191us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:50.830827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:50.830829 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 02:57:50.830897 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:50.830899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:50.831505 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:50.831506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:50.832091 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:50.832094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:50.832420 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:50.832424 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:50.832486 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:50.832505 (+ 19us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 02:57:50.853239 (+ 20734us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0924 02:57:50.853356 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:36969: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":64,"threads_started":1}
E20260924 02:57:50.853699 5586 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260924 02:57:50.853818 5586 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.0.98.62:36969: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43543/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260924 02:57:50.854216 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 5489
2026-09-24T02:57:50Z chronyd exiting
I20260924 02:57:50.889321 392 test_util.cc:175] -----------------------------------------------
I20260924 02:57:50.889395 392 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790218647334776-392-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (218 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260924 02:57:50.890097 392 mini_oidc.cc:152] Starting JWKS server
I20260924 02:57:50.890797 392 webserver.cc:460] Webserver started at https://127.0.0.1:40719/ using document root <none> and password file <none>
I20260924 02:57:50.890848 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 02:57:50.890981 392 webserver.cc:460] Webserver started at http://127.0.0.1:42235/ using document root <none> and password file <none>
2026-09-24T02:57:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:57:50Z Disabled control of system clock
I20260924 02:57:50.902563 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40939
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:45393
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40939
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40719/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/testchainca.cert with env {}
W20260924 02:57:50.979971 5592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:50.980170 5592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:50.980189 5592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:50.981340 5592 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:50.981379 5592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:50.981390 5592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:50.981412 5592 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:50.981436 5592 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 02:57:50.981456 5592 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 02:57:50.981480 5592 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:40719/jwks.json
I20260924 02:57:50.982995 5592 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45393
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40939
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40939
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:40719/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:50.983210 5592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:50.983392 5592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:50.987839 5597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:50.987864 5600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:50.987881 5598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:50.988034 5592 server_base.cc:1034] running on GCE node
I20260924 02:57:50.988255 5592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:50.988487 5592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:50.989616 5592 hybrid_clock.cc:648] HybridClock initialized: now 1790218670989602 us; error 28 us; skew 500 ppm
I20260924 02:57:50.994258 5592 webserver.cc:460] Webserver started at http://127.0.98.62:41653/ using document root <none> and password file <none>
I20260924 02:57:50.994467 5592 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:50.994513 5592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:50.994660 5592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:50.995666 5592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "2da80da1868a43248bad360f7792a723"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "ecde76b0f396bf947c910b1b5bb5b923"
server_key_iv: "ad8dd5e99beb8dabfabb91b3b38dd62c"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.995990 5592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "2da80da1868a43248bad360f7792a723"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "ecde76b0f396bf947c910b1b5bb5b923"
server_key_iv: "ad8dd5e99beb8dabfabb91b3b38dd62c"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.997108 5592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 02:57:50.997712 5606 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:50.997861 5592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:50.997938 5592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "2da80da1868a43248bad360f7792a723"
format_stamp: "Formatted at 2026-09-24 02:57:50 on dist-test-slave-sb6p"
server_key: "ecde76b0f396bf947c910b1b5bb5b923"
server_key_iv: "ad8dd5e99beb8dabfabb91b3b38dd62c"
server_key_version: "encryptionkey@0"
I20260924 02:57:50.998001 5592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:51.015260 5592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:51.015579 5592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:51.015708 5592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:51.019218 5592 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40939
I20260924 02:57:51.019253 5658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40939 every 8 connection(s)
I20260924 02:57:51.019570 5592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:57:51.020514 5659 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:51.022776 5659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723: Bootstrap starting.
I20260924 02:57:51.023284 5659 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:51.023538 5659 log.cc:824] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:51.024183 5659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723: No bootstrap required, opened a new log
I20260924 02:57:51.025703 5659 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da80da1868a43248bad360f7792a723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260924 02:57:51.025841 5659 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:51.025862 5659 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2da80da1868a43248bad360f7792a723, State: Initialized, Role: FOLLOWER
I20260924 02:57:51.025969 5659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da80da1868a43248bad360f7792a723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260924 02:57:51.026038 5659 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:57:51.026067 5659 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:57:51.026098 5659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:51.026551 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 5592
I20260924 02:57:51.026654 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:57:51.026690 5659 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da80da1868a43248bad360f7792a723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260924 02:57:51.026793 5659 leader_election.cc:302] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2da80da1868a43248bad360f7792a723; no voters:
I20260924 02:57:51.026836 392 external_mini_cluster.cc:1428] Setting key c6f45c9ad9bc95be56bb2131719f9309
I20260924 02:57:51.026947 5659 leader_election.cc:288] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:57:51.027011 5662 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:51.027185 5662 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [term 1 LEADER]: Becoming Leader. State: Replica: 2da80da1868a43248bad360f7792a723, State: Running, Role: LEADER
I20260924 02:57:51.027408 5662 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da80da1868a43248bad360f7792a723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } }
I20260924 02:57:51.027189 5659 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:51.027736 5664 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2da80da1868a43248bad360f7792a723. Latest consensus state: current_term: 1 leader_uuid: "2da80da1868a43248bad360f7792a723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da80da1868a43248bad360f7792a723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } } }
I20260924 02:57:51.027812 5664 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:51.027840 5663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2da80da1868a43248bad360f7792a723" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2da80da1868a43248bad360f7792a723" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40939 } } }
I20260924 02:57:51.027909 5663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:51.028367 5668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:51.028862 5668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:57:51.030303 5668 catalog_manager.cc:1347] Generated new cluster ID: 945eee6bbb344ebfa4b10f2226ae64a3
I20260924 02:57:51.030352 5668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:57:51.032895 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:51.027301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47330 (local address 127.0.98.62:40939)
0924 02:57:51.027502 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:51.027511 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:51.027542 (+ 31us) server_negotiation.cc:407] Connection header received
0924 02:57:51.027645 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:51.027652 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:51.027956 (+ 304us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:51.028372 (+ 416us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:51.028476 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.029820 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.030889 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.031479 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.031585 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:51.032259 (+ 674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:51.032262 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:51.032262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:57:51.032440 (+ 178us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:51.032476 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:57:51.032488 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:51.032540 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:51.032624 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 02:57:51.032697 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":57,"threads_started":1}
I20260924 02:57:51.033766 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 02:57:51.058032 5668 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:57:51.058640 5668 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:51.064666 5668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2da80da1868a43248bad360f7792a723: Generated new TSK 0
I20260924 02:57:51.064841 5668 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260924 02:57:51.084605 5689 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260924 02:57:51.084685 5689 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260924 02:57:51.084991 5666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:51.071507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47346 (local address 127.0.98.62:40939)
0924 02:57:51.071552 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:51.071554 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:51.071563 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:57:51.071630 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:51.071631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:51.071643 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:51.071712 (+ 69us) server_negotiation.cc:226] Negotiated authn=JWT
0924 02:57:51.071840 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.072327 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.073077 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.073277 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.073337 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:51.073488 (+ 151us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 02:57:51.084779 (+ 11291us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40719/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 02:57:51.084922 (+ 143us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47346: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40719/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":24}
W20260924 02:57:51.085049 5666 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47346: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40719/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260924 02:57:51.085083 5688 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:57:51.071369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40939 (local address 127.0.0.1:47346)
0924 02:57:51.071496 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:51.071507 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:51.071562 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:51.071716 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:51.071717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:51.071719 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 02:57:51.071783 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:51.071785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.072413 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.072414 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:51.073005 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:51.073007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.073336 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.073339 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:51.073398 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:51.073403 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 02:57:51.084893 (+ 11490us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40719/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0924 02:57:51.084997 (+ 104us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:40939: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:40719/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260924 02:57:51.085414 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 5592
2026-09-24T02:57:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (231 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260924 02:57:51.121870 392 mini_oidc.cc:152] Starting JWKS server
I20260924 02:57:51.122510 392 webserver.cc:460] Webserver started at https://127.0.0.1:36107/ using document root <none> and password file <none>
I20260924 02:57:51.122586 392 mini_oidc.cc:189] Starting OIDC Discovery server
I20260924 02:57:51.122720 392 webserver.cc:460] Webserver started at http://127.0.0.1:43631/ using document root <none> and password file <none>
2026-09-24T02:57:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:57:51Z Disabled control of system clock
I20260924 02:57:51.129317 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38085
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:39023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38085
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36107/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 02:57:51.206116 5695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:51.206342 5695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:51.206372 5695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:51.207551 5695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:51.207593 5695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:51.207609 5695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:51.207626 5695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:51.207641 5695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 02:57:51.207659 5695 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 02:57:51.207681 5695 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36107/jwks.json
I20260924 02:57:51.209043 5695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38085
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36107/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5695
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:51.209282 5695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:51.209544 5695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:51.214015 5703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:51.213975 5701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:51.214119 5695 server_base.cc:1034] running on GCE node
W20260924 02:57:51.213989 5700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:51.214383 5695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:51.214671 5695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:51.215811 5695 hybrid_clock.cc:648] HybridClock initialized: now 1790218671215793 us; error 33 us; skew 500 ppm
I20260924 02:57:51.216964 5695 webserver.cc:460] Webserver started at http://127.0.98.62:38751/ using document root <none> and password file <none>
I20260924 02:57:51.217157 5695 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:51.217200 5695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:51.217315 5695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:51.218335 5695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "abfdf2151bb549788bb680d81b58dffb"
format_stamp: "Formatted at 2026-09-24 02:57:51 on dist-test-slave-sb6p"
server_key: "6f4834cfb4b8c25b8acddd6b6b1740dd"
server_key_iv: "16fa7d1272bb6e80afd428997c7fa8ee"
server_key_version: "encryptionkey@0"
I20260924 02:57:51.218663 5695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "abfdf2151bb549788bb680d81b58dffb"
format_stamp: "Formatted at 2026-09-24 02:57:51 on dist-test-slave-sb6p"
server_key: "6f4834cfb4b8c25b8acddd6b6b1740dd"
server_key_iv: "16fa7d1272bb6e80afd428997c7fa8ee"
server_key_version: "encryptionkey@0"
I20260924 02:57:51.219869 5695 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 02:57:51.220568 5709 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:51.220719 5695 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:51.220799 5695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "abfdf2151bb549788bb680d81b58dffb"
format_stamp: "Formatted at 2026-09-24 02:57:51 on dist-test-slave-sb6p"
server_key: "6f4834cfb4b8c25b8acddd6b6b1740dd"
server_key_iv: "16fa7d1272bb6e80afd428997c7fa8ee"
server_key_version: "encryptionkey@0"
I20260924 02:57:51.220858 5695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:51.233848 5695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:51.234112 5695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:51.234234 5695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:51.237689 5695 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38085
I20260924 02:57:51.237720 5761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38085 every 8 connection(s)
I20260924 02:57:51.238003 5695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:57:51.238631 5762 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:51.241079 5762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb: Bootstrap starting.
I20260924 02:57:51.241616 5762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:51.241873 5762 log.cc:824] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:51.242561 5762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb: No bootstrap required, opened a new log
I20260924 02:57:51.242791 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 5695
I20260924 02:57:51.242875 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:57:51.243031 392 external_mini_cluster.cc:1428] Setting key 45621ee59e92e871a0e7f741413d6af7
I20260924 02:57:51.244285 5762 raft_consensus.cc:348] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfdf2151bb549788bb680d81b58dffb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38085 } }
I20260924 02:57:51.244429 5762 raft_consensus.cc:374] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:51.244462 5762 raft_consensus.cc:729] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: abfdf2151bb549788bb680d81b58dffb, State: Initialized, Role: FOLLOWER
I20260924 02:57:51.244585 5762 consensus_queue.cc:260] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfdf2151bb549788bb680d81b58dffb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38085 } }
I20260924 02:57:51.244652 5762 raft_consensus.cc:388] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:57:51.244681 5762 raft_consensus.cc:482] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:57:51.244722 5762 raft_consensus.cc:3037] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:51.245217 5762 raft_consensus.cc:504] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfdf2151bb549788bb680d81b58dffb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38085 } }
I20260924 02:57:51.245306 5762 leader_election.cc:302] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: abfdf2151bb549788bb680d81b58dffb; no voters:
I20260924 02:57:51.245460 5762 leader_election.cc:288] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:57:51.245493 5767 raft_consensus.cc:2781] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:51.245589 5767 raft_consensus.cc:686] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [term 1 LEADER]: Becoming Leader. State: Replica: abfdf2151bb549788bb680d81b58dffb, State: Running, Role: LEADER
I20260924 02:57:51.245673 5767 consensus_queue.cc:237] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfdf2151bb549788bb680d81b58dffb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38085 } }
I20260924 02:57:51.245702 5762 sys_catalog.cc:564] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:51.246105 5769 sys_catalog.cc:455] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [sys.catalog]: SysCatalogTable state changed. Reason: New leader abfdf2151bb549788bb680d81b58dffb. Latest consensus state: current_term: 1 leader_uuid: "abfdf2151bb549788bb680d81b58dffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfdf2151bb549788bb680d81b58dffb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38085 } } }
I20260924 02:57:51.246099 5768 sys_catalog.cc:455] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "abfdf2151bb549788bb680d81b58dffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "abfdf2151bb549788bb680d81b58dffb" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38085 } } }
I20260924 02:57:51.246206 5769 sys_catalog.cc:458] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:51.246230 5768 sys_catalog.cc:458] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:51.246446 5772 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:51.247223 5772 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:57:51.249398 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:51.243496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56654 (local address 127.0.98.62:38085)
0924 02:57:51.243700 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:51.243707 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:51.243735 (+ 28us) server_negotiation.cc:407] Connection header received
0924 02:57:51.244309 (+ 574us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:51.244317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:51.244384 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:51.244523 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:51.244782 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.246396 (+ 1614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.247548 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.248043 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.248182 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:51.248783 (+ 601us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:51.248786 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:51.248787 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:57:51.248944 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:51.248977 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:57:51.248987 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:51.249039 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:51.249109 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 02:57:51.249189 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":80,"threads_started":1}
I20260924 02:57:51.250257 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 02:57:51.250912 5772 catalog_manager.cc:1347] Generated new cluster ID: cb95ee4da69e4b1ba68712d0b340d9a2
I20260924 02:57:51.250965 5772 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:57:51.259863 5772 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:57:51.260560 5772 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:51.266045 5772 catalog_manager.cc:5868] T 00000000000000000000000000000000 P abfdf2151bb549788bb680d81b58dffb: Generated new TSK 0
I20260924 02:57:51.266225 5772 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 02:57:51.277292 5766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:51.273468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56670 (local address 127.0.98.62:38085)
0924 02:57:51.273515 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:51.273517 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:51.273577 (+ 60us) server_negotiation.cc:407] Connection header received
0924 02:57:51.273659 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:51.273660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:51.273671 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:51.273741 (+ 70us) server_negotiation.cc:226] Negotiated authn=JWT
0924 02:57:51.273918 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.274711 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.275567 (+ 856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.275892 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.275962 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:51.276147 (+ 185us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 02:57:51.277092 (+ 945us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36107/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 02:57:51.277222 (+ 130us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56670: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36107/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 02:57:51.277350 5766 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56670: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36107/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260924 02:57:51.279261 5791 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:57:51.273393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38085 (local address 127.0.0.1:56670)
0924 02:57:51.273519 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:51.273529 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:51.273583 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:51.273785 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:51.273786 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:51.273788 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 02:57:51.273842 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:51.273844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.274806 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.274807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:51.275461 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:51.275464 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.275990 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.275994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:51.276045 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:51.276050 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 02:57:51.277187 (+ 1137us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36107/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0924 02:57:51.279163 (+ 1976us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:38085: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36107/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":1794,"thread_start_us":74,"threads_started":1}
I20260924 02:57:51.279495 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 5695
E20260924 02:57:51.288329 5792 webserver.cc:533] Webserver: sslize failed:
E20260924 02:57:51.288393 5792 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-24T02:57:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (193 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-24T02:57:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:57:51Z Disabled control of system clock
I20260924 02:57:51.322340 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35339
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:34727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35339
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/testchainca.cert with env {}
W20260924 02:57:51.399034 5796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:51.399231 5796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:51.399252 5796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:51.400431 5796 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:51.400465 5796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:51.400476 5796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:51.400487 5796 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:51.400497 5796 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260924 02:57:51.400516 5796 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260924 02:57:51.400527 5796 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260924 02:57:51.401942 5796 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35339
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35339
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:51.402129 5796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:51.402287 5796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:51.406817 5801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:51.406945 5796 server_base.cc:1034] running on GCE node
W20260924 02:57:51.406857 5802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:51.406849 5804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:51.407241 5796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:51.407498 5796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:51.408631 5796 hybrid_clock.cc:648] HybridClock initialized: now 1790218671408619 us; error 29 us; skew 500 ppm
I20260924 02:57:51.409796 5796 webserver.cc:460] Webserver started at http://127.0.98.62:35905/ using document root <none> and password file <none>
I20260924 02:57:51.409999 5796 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:51.410044 5796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:51.410167 5796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:51.411224 5796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "8476d5b0ba534e8d993d006a786d5ce0"
format_stamp: "Formatted at 2026-09-24 02:57:51 on dist-test-slave-sb6p"
server_key: "30dc72caec8da086f3a3da0ca4672bf0"
server_key_iv: "17628b611abed61805c5d2e0a7755781"
server_key_version: "encryptionkey@0"
I20260924 02:57:51.411553 5796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "8476d5b0ba534e8d993d006a786d5ce0"
format_stamp: "Formatted at 2026-09-24 02:57:51 on dist-test-slave-sb6p"
server_key: "30dc72caec8da086f3a3da0ca4672bf0"
server_key_iv: "17628b611abed61805c5d2e0a7755781"
server_key_version: "encryptionkey@0"
I20260924 02:57:51.412695 5796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:57:51.413311 5810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:51.413444 5796 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:51.413520 5796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "8476d5b0ba534e8d993d006a786d5ce0"
format_stamp: "Formatted at 2026-09-24 02:57:51 on dist-test-slave-sb6p"
server_key: "30dc72caec8da086f3a3da0ca4672bf0"
server_key_iv: "17628b611abed61805c5d2e0a7755781"
server_key_version: "encryptionkey@0"
I20260924 02:57:51.413583 5796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:51.434351 5796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:51.434687 5796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:51.434816 5796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:51.438298 5796 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35339
I20260924 02:57:51.438442 5862 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35339 every 8 connection(s)
I20260924 02:57:51.438714 5796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:57:51.439607 5863 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:51.441856 5863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0: Bootstrap starting.
I20260924 02:57:51.442394 5863 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:51.442682 5863 log.cc:824] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:51.443267 5863 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0: No bootstrap required, opened a new log
I20260924 02:57:51.444768 5863 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476d5b0ba534e8d993d006a786d5ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } }
I20260924 02:57:51.444902 5863 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:51.444924 5863 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8476d5b0ba534e8d993d006a786d5ce0, State: Initialized, Role: FOLLOWER
I20260924 02:57:51.445060 5863 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476d5b0ba534e8d993d006a786d5ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } }
I20260924 02:57:51.445118 5863 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:57:51.445139 5863 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:57:51.445192 5863 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:51.445664 5863 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476d5b0ba534e8d993d006a786d5ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } }
I20260924 02:57:51.445748 5863 leader_election.cc:302] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8476d5b0ba534e8d993d006a786d5ce0; no voters:
I20260924 02:57:51.445906 5863 leader_election.cc:288] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:57:51.445951 5866 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:51.446009 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 5796
I20260924 02:57:51.446107 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:57:51.446110 5866 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [term 1 LEADER]: Becoming Leader. State: Replica: 8476d5b0ba534e8d993d006a786d5ce0, State: Running, Role: LEADER
I20260924 02:57:51.446170 5863 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:51.446235 5866 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476d5b0ba534e8d993d006a786d5ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } }
I20260924 02:57:51.446313 392 external_mini_cluster.cc:1428] Setting key 1af658e0c6a78aacd989f0268e4d01da
I20260924 02:57:51.446894 5868 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8476d5b0ba534e8d993d006a786d5ce0. Latest consensus state: current_term: 1 leader_uuid: "8476d5b0ba534e8d993d006a786d5ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476d5b0ba534e8d993d006a786d5ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } }
I20260924 02:57:51.446976 5868 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:51.447050 5867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8476d5b0ba534e8d993d006a786d5ce0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8476d5b0ba534e8d993d006a786d5ce0" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35339 } } }
I20260924 02:57:51.447172 5867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:51.447165 5877 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:51.447683 5877 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:57:51.449205 5877 catalog_manager.cc:1347] Generated new cluster ID: cbba6dedccdb4a27b26f56372cda25c9
I20260924 02:57:51.449254 5877 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:57:51.452553 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:51.446784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.0.98.62:35339)
0924 02:57:51.446972 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:51.446981 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:51.447340 (+ 359us) server_negotiation.cc:407] Connection header received
0924 02:57:51.447496 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:51.447503 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:51.447572 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:51.447708 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:51.448084 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.449412 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.450461 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.451127 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.451255 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:51.451874 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:51.451877 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:51.451878 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:57:51.452081 (+ 203us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:51.452124 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:57:51.452139 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:51.452207 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:51.452262 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 02:57:51.452339 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":71,"threads_started":1}
I20260924 02:57:51.453502 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 02:57:51.456688 5877 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:57:51.457167 5877 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:51.461522 5877 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8476d5b0ba534e8d993d006a786d5ce0: Generated new TSK 0
I20260924 02:57:51.461700 5877 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 02:57:51.529090 5876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:51.467710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56340 (local address 127.0.98.62:35339)
0924 02:57:51.467751 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:51.467753 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:51.467809 (+ 56us) server_negotiation.cc:407] Connection header received
0924 02:57:51.467877 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:51.467878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:51.467889 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:51.467968 (+ 79us) server_negotiation.cc:226] Negotiated authn=JWT
0924 02:57:51.468058 (+ 90us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.468818 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.469532 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.469861 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.469928 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:51.470052 (+ 124us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0924 02:57:51.528803 (+ 58751us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 02:57:51.529008 (+ 205us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56340: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":23}
W20260924 02:57:51.529162 5892 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:57:51.467628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:35339 (local address 127.0.0.1:56340)
0924 02:57:51.467743 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:51.467753 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:51.467809 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:51.467967 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:51.467967 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:51.467969 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0924 02:57:51.468012 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:51.468013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.468902 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.468902 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:51.469463 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:51.469465 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:51.469929 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:51.469932 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:51.469971 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:51.469976 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0924 02:57:51.528959 (+ 58983us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0924 02:57:51.529086 (+ 127us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.98.62:35339: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":71,"threads_started":1}
W20260924 02:57:51.529189 5876 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56340: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
I20260924 02:57:51.529425 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 5796
2026-09-24T02:57:51Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (229 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (83 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (84 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:57:51 dist-test-slave-sb6p krb5kdc[5904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:57:51 dist-test-slave-sb6p krb5kdc[5904](info): set up 2 sockets
Sep 24 02:57:51 dist-test-slave-sb6p krb5kdc[5904](info): commencing operation
krb5kdc: starting...
W20260924 02:57:52.650401 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:57:52 dist-test-slave-sb6p krb5kdc[5904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:57:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:57:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 02:57:52.758916 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34853
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34853
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:52.835000 5917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:52.835188 5917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:52.835208 5917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:52.836304 5917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:52.836337 5917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:52.836359 5917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:52.836372 5917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:52.836398 5917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:57:52.837839 5917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34853
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34853
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:52.838018 5917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:52.838182 5917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:52.842722 5922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:52.842762 5917 server_base.cc:1034] running on GCE node
W20260924 02:57:52.842730 5925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:52.842737 5923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:52.843127 5917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:52.843393 5917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:52.844529 5917 hybrid_clock.cc:648] HybridClock initialized: now 1790218672844513 us; error 27 us; skew 500 ppm
Sep 24 02:57:52 dist-test-slave-sb6p krb5kdc[5904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218672, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:52.846804 5917 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:57:52.847175 5917 webserver.cc:460] Webserver started at http://127.0.98.62:46521/ using document root <none> and password file <none>
I20260924 02:57:52.847378 5917 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:52.847424 5917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:52.847541 5917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:52.848536 5917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "f7b09235299849b9bee215fe87210069"
format_stamp: "Formatted at 2026-09-24 02:57:52 on dist-test-slave-sb6p"
server_key: "8953610bc68eefe330a7407ec0688e21"
server_key_iv: "e0f028007270618b67ff1e23be254588"
server_key_version: "encryptionkey@0"
I20260924 02:57:52.848838 5917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "f7b09235299849b9bee215fe87210069"
format_stamp: "Formatted at 2026-09-24 02:57:52 on dist-test-slave-sb6p"
server_key: "8953610bc68eefe330a7407ec0688e21"
server_key_iv: "e0f028007270618b67ff1e23be254588"
server_key_version: "encryptionkey@0"
I20260924 02:57:52.849983 5917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:57:52.850620 5932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:52.850759 5917 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:52.850837 5917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "f7b09235299849b9bee215fe87210069"
format_stamp: "Formatted at 2026-09-24 02:57:52 on dist-test-slave-sb6p"
server_key: "8953610bc68eefe330a7407ec0688e21"
server_key_iv: "e0f028007270618b67ff1e23be254588"
server_key_version: "encryptionkey@0"
I20260924 02:57:52.850903 5917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:52.879617 5917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:52.881315 5917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:52.881450 5917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:52.885027 5917 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34853
I20260924 02:57:52.885070 5984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34853 every 8 connection(s)
I20260924 02:57:52.885533 5917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:57:52.886168 5985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:52.888377 5985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069: Bootstrap starting.
I20260924 02:57:52.888911 5985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:52.889164 5985 log.cc:824] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:52.889748 5985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069: No bootstrap required, opened a new log
I20260924 02:57:52.891320 5985 raft_consensus.cc:348] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b09235299849b9bee215fe87210069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34853 } }
I20260924 02:57:52.891460 5985 raft_consensus.cc:374] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:52.891491 5985 raft_consensus.cc:729] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f7b09235299849b9bee215fe87210069, State: Initialized, Role: FOLLOWER
I20260924 02:57:52.891623 5985 consensus_queue.cc:260] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b09235299849b9bee215fe87210069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34853 } }
I20260924 02:57:52.891688 5985 raft_consensus.cc:388] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:57:52.891711 5985 raft_consensus.cc:482] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:57:52.891755 5985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:52.892561 5985 raft_consensus.cc:504] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b09235299849b9bee215fe87210069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34853 } }
I20260924 02:57:52.892652 5985 leader_election.cc:302] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f7b09235299849b9bee215fe87210069; no voters:
I20260924 02:57:52.892795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 5917
I20260924 02:57:52.892815 5985 leader_election.cc:288] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:57:52.892854 5988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:52.892900 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:57:52.893047 5988 raft_consensus.cc:686] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [term 1 LEADER]: Becoming Leader. State: Replica: f7b09235299849b9bee215fe87210069, State: Running, Role: LEADER
I20260924 02:57:52.893090 392 external_mini_cluster.cc:1428] Setting key a3794b21eca4c5c91a8d6a54ea42a40b
I20260924 02:57:52.893055 5985 sys_catalog.cc:564] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:52.893198 5988 consensus_queue.cc:237] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b09235299849b9bee215fe87210069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34853 } }
I20260924 02:57:52.893565 5989 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f7b09235299849b9bee215fe87210069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b09235299849b9bee215fe87210069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34853 } } }
I20260924 02:57:52.893642 5989 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:52.893813 5990 sys_catalog.cc:455] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f7b09235299849b9bee215fe87210069. Latest consensus state: current_term: 1 leader_uuid: "f7b09235299849b9bee215fe87210069" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f7b09235299849b9bee215fe87210069" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34853 } } }
I20260924 02:57:52.893891 5990 sys_catalog.cc:458] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:52.893987 5998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:52.894483 5998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:57:52.896276 5998 catalog_manager.cc:1347] Generated new cluster ID: eed3ec37102043c89fc3198be02a9ef0
I20260924 02:57:52.896329 5998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 02:57:52 dist-test-slave-sb6p krb5kdc[5904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:52.903342 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:52.893544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50722 (local address 127.0.98.62:34853)
0924 02:57:52.893700 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:52.893709 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:52.894253 (+ 544us) server_negotiation.cc:407] Connection header received
0924 02:57:52.894382 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:52.894389 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:52.894456 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:52.895272 (+ 816us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:52.895378 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:52.896563 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:52.897278 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:52.897695 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:52.897792 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:52.899950 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:52.899954 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:52.899954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:52.899963 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:52.901875 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:52.902176 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:52.902180 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:52.902180 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:52.902258 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:52.902434 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:52.902435 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:52.902435 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:52.902698 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:52.902847 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:52.902915 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 02:57:52.903084 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":66,"threads_started":1}
I20260924 02:57:52.907001 5998 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:57:52.907491 5998 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:52.917069 5998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f7b09235299849b9bee215fe87210069: Generated new TSK 0
I20260924 02:57:52.917240 5998 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 02:57:52.941505 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34853
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:53.019312 6011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:53.019543 6011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:53.019567 6011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:53.020742 6011 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:53.020783 6011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:53.020808 6011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:57:53.022238 6011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6011
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:53.022439 6011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:53.022697 6011 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:53.027661 6016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:53.027661 6017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:53.027683 6019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:53.027931 6011 server_base.cc:1034] running on GCE node
I20260924 02:57:53.028137 6011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:53.028357 6011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:53.029505 6011 hybrid_clock.cc:648] HybridClock initialized: now 1790218673029496 us; error 33 us; skew 500 ppm
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[5904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:53.031948 6011 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 02:57:53.032383 6011 webserver.cc:460] Webserver started at http://127.0.98.1:44283/ using document root <none> and password file <none>
I20260924 02:57:53.032586 6011 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:53.032642 6011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:53.032737 6011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:53.033712 6011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "362518c5e4ba45cfb262c41efb43807a"
format_stamp: "Formatted at 2026-09-24 02:57:53 on dist-test-slave-sb6p"
server_key: "88bb84c0894d370c2d366f4e8f3c74b1"
server_key_iv: "d2c6ce11ecab36a82ad0593f0307a56f"
server_key_version: "encryptionkey@0"
I20260924 02:57:53.034004 6011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "362518c5e4ba45cfb262c41efb43807a"
format_stamp: "Formatted at 2026-09-24 02:57:53 on dist-test-slave-sb6p"
server_key: "88bb84c0894d370c2d366f4e8f3c74b1"
server_key_iv: "d2c6ce11ecab36a82ad0593f0307a56f"
server_key_version: "encryptionkey@0"
I20260924 02:57:53.035271 6011 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 02:57:53.035917 6026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:53.036055 6011 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:53.036134 6011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "362518c5e4ba45cfb262c41efb43807a"
format_stamp: "Formatted at 2026-09-24 02:57:53 on dist-test-slave-sb6p"
server_key: "88bb84c0894d370c2d366f4e8f3c74b1"
server_key_iv: "d2c6ce11ecab36a82ad0593f0307a56f"
server_key_version: "encryptionkey@0"
I20260924 02:57:53.036197 6011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:53.053753 6011 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:53.055617 6011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:53.055737 6011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:53.055965 6011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:53.056252 6011 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:53.056283 6011 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:53.056306 6011 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:53.056329 6011 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:53.061746 6011 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:46467
I20260924 02:57:53.061798 6139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:46467 every 8 connection(s)
I20260924 02:57:53.062085 6011 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:57:53.065749 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 6011
I20260924 02:57:53.065836 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:57:53.066000 392 external_mini_cluster.cc:1428] Setting key a291aeeaa3671d26071c4564a5165e9b
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[5904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:53.071275 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:53.062769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60451 (local address 127.0.98.62:34853)
0924 02:57:53.062826 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:53.062828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:53.063002 (+ 174us) server_negotiation.cc:407] Connection header received
0924 02:57:53.063919 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:53.063921 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:53.063932 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:53.063984 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:53.064592 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.065097 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.066136 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.066377 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.066450 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:53.068508 (+ 2058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:53.068509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:53.068509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:53.068519 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:53.069970 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:53.070333 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:53.070335 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:53.070336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:53.070400 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:53.070646 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:53.070646 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:53.070647 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:53.070788 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:53.070858 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:53.071084 (+ 226us) server_negotiation.cc:299] Negotiation successful
0924 02:57:53.071178 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 02:57:53.071347 6142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:53.062736 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34853 (local address 127.0.98.1:60451)
0924 02:57:53.062874 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:53.062920 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:53.063811 (+ 891us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:53.064053 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:53.064057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:53.064308 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:53.064496 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:53.064500 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.065185 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.065186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:53.066028 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:53.066031 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.066459 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.066460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:53.066623 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:53.067090 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:53.067100 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:53.068416 (+ 1316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:53.070043 (+ 1627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:53.070049 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:53.070051 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:53.070261 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:53.070461 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:53.070462 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:53.070462 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:53.070588 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:53.070846 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:53.070849 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:53.070967 (+ 118us) client_negotiation.cc:769] Sending connection context
0924 02:57:53.071045 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 02:57:53.071165 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260924 02:57:53.071867 6140 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34853
I20260924 02:57:53.071957 6140 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:53.072124 6140 heartbeater.cc:511] Master 127.0.98.62:34853 requested a full tablet report, sending...
I20260924 02:57:53.072679 5949 ts_manager.cc:194] Registered new tserver with Master: 362518c5e4ba45cfb262c41efb43807a (127.0.98.1:46467)
I20260924 02:57:53.073582 5949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60451
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 02:57:53.089313 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34853
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:53.167649 6145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:53.167850 6145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:53.167873 6145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:53.169030 6145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:53.169068 6145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:53.169097 6145 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:57:53.170506 6145 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:53.170734 6145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:53.170935 6145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:53.175465 6150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:53.175482 6153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:53.175550 6145 server_base.cc:1034] running on GCE node
W20260924 02:57:53.175490 6151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:53.175890 6145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:53.176106 6145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:53.177237 6145 hybrid_clock.cc:648] HybridClock initialized: now 1790218673177227 us; error 26 us; skew 500 ppm
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[5904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:53.179452 6145 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 02:57:53.179878 6145 webserver.cc:460] Webserver started at http://127.0.98.2:40171/ using document root <none> and password file <none>
I20260924 02:57:53.180063 6145 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:53.180112 6145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:53.180230 6145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:53.181195 6145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "e1cd000e53dd40dbbc4a48fa91f1518b"
format_stamp: "Formatted at 2026-09-24 02:57:53 on dist-test-slave-sb6p"
server_key: "66dcf55ab5f0fd2c8d050eca703bc19c"
server_key_iv: "00dce1abae86050f9209b6e51bfd5af2"
server_key_version: "encryptionkey@0"
I20260924 02:57:53.181497 6145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e1cd000e53dd40dbbc4a48fa91f1518b"
format_stamp: "Formatted at 2026-09-24 02:57:53 on dist-test-slave-sb6p"
server_key: "66dcf55ab5f0fd2c8d050eca703bc19c"
server_key_iv: "00dce1abae86050f9209b6e51bfd5af2"
server_key_version: "encryptionkey@0"
I20260924 02:57:53.182636 6145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 02:57:53.183303 6160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:53.183445 6145 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:53.183524 6145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "e1cd000e53dd40dbbc4a48fa91f1518b"
format_stamp: "Formatted at 2026-09-24 02:57:53 on dist-test-slave-sb6p"
server_key: "66dcf55ab5f0fd2c8d050eca703bc19c"
server_key_iv: "00dce1abae86050f9209b6e51bfd5af2"
server_key_version: "encryptionkey@0"
I20260924 02:57:53.183583 6145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:53.210760 6145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:53.212529 6145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:53.212663 6145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:53.212890 6145 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:53.213177 6145 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:53.213209 6145 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:53.213238 6145 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:53.213258 6145 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:53.218562 6145 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43647
I20260924 02:57:53.218617 6273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43647 every 8 connection(s)
I20260924 02:57:53.218911 6145 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 02:57:53.223639 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 6145
I20260924 02:57:53.223733 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:57:53.223888 392 external_mini_cluster.cc:1428] Setting key 4cf6df709fdad706a72f24e05a11ebb6
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[5904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:53.227846 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:53.219598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55273 (local address 127.0.98.62:34853)
0924 02:57:53.219677 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:53.219679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:53.219811 (+ 132us) server_negotiation.cc:407] Connection header received
0924 02:57:53.220724 (+ 913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:53.220726 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:53.220735 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:53.220786 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:53.221364 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.221862 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.222776 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.222973 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.223020 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:53.225079 (+ 2059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:53.225080 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:53.225080 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:53.225088 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:53.226556 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:53.226999 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:53.227000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:53.227000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:53.227047 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:53.227277 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:53.227277 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:53.227277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:53.227372 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:53.227415 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:53.227699 (+ 284us) server_negotiation.cc:299] Negotiation successful
0924 02:57:53.227765 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 02:57:53.228210 6276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:53.219565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34853 (local address 127.0.98.2:55273)
0924 02:57:53.219702 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:53.219736 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:53.220634 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:53.220871 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:53.220875 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:53.221114 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:53.221301 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:53.221305 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.221920 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.221921 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:53.222722 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:53.222724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.223028 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.223029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:53.223125 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:53.223535 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:53.223543 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:53.224992 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:53.226623 (+ 1631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:53.226630 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:53.226633 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:53.226930 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:53.227096 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:53.227096 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:53.227097 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:53.227220 (+ 123us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:53.227420 (+ 200us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:53.227424 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:53.227577 (+ 153us) client_negotiation.cc:769] Sending connection context
0924 02:57:53.227690 (+ 113us) client_negotiation.cc:240] Negotiation successful
0924 02:57:53.227867 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
I20260924 02:57:53.228688 6274 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34853
I20260924 02:57:53.228796 6274 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:53.228991 6274 heartbeater.cc:511] Master 127.0.98.62:34853 requested a full tablet report, sending...
I20260924 02:57:53.229367 5949 ts_manager.cc:194] Registered new tserver with Master: e1cd000e53dd40dbbc4a48fa91f1518b (127.0.98.2:43647)
I20260924 02:57:53.229718 5949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55273
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 02:57:53.247324 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34853
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:53.327162 6279 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:53.327368 6279 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:53.327389 6279 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:53.328519 6279 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:53.328558 6279 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:53.328594 6279 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:57:53.330118 6279 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34853
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6279
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:53.330355 6279 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:53.330585 6279 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:53.335151 6285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:53.335125 6284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:53.335386 6279 server_base.cc:1034] running on GCE node
W20260924 02:57:53.335134 6287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:53.335642 6279 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:53.335875 6279 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:53.337025 6279 hybrid_clock.cc:648] HybridClock initialized: now 1790218673337013 us; error 28 us; skew 500 ppm
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[5904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:53.339496 6279 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 02:57:53.340039 6279 webserver.cc:460] Webserver started at http://127.0.98.3:46379/ using document root <none> and password file <none>
I20260924 02:57:53.340260 6279 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:53.340305 6279 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:53.340426 6279 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:53.341398 6279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "3ce8abd350504f53a0a490a9b491f09f"
format_stamp: "Formatted at 2026-09-24 02:57:53 on dist-test-slave-sb6p"
server_key: "bcecd7e352778cb9c87ce04617eedbe9"
server_key_iv: "0b01abb7be0fa62dedff91c9a5990bac"
server_key_version: "encryptionkey@0"
I20260924 02:57:53.341715 6279 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3ce8abd350504f53a0a490a9b491f09f"
format_stamp: "Formatted at 2026-09-24 02:57:53 on dist-test-slave-sb6p"
server_key: "bcecd7e352778cb9c87ce04617eedbe9"
server_key_iv: "0b01abb7be0fa62dedff91c9a5990bac"
server_key_version: "encryptionkey@0"
I20260924 02:57:53.342906 6279 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 02:57:53.343560 6294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:53.343716 6279 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:53.343794 6279 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "3ce8abd350504f53a0a490a9b491f09f"
format_stamp: "Formatted at 2026-09-24 02:57:53 on dist-test-slave-sb6p"
server_key: "bcecd7e352778cb9c87ce04617eedbe9"
server_key_iv: "0b01abb7be0fa62dedff91c9a5990bac"
server_key_version: "encryptionkey@0"
I20260924 02:57:53.343856 6279 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:53.362306 6279 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:53.364259 6279 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:53.364390 6279 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:53.364588 6279 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:53.364856 6279 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:53.364886 6279 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:53.364913 6279 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:53.364930 6279 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:53.370128 6279 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42231
I20260924 02:57:53.370190 6407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42231 every 8 connection(s)
I20260924 02:57:53.370453 6279 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 02:57:53.371539 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 6279
I20260924 02:57:53.371618 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:57:53.371771 392 external_mini_cluster.cc:1428] Setting key 96c6fdc9785da693e256ca6c3dc4f1c3
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[5904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:57:53.378774 5995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:53.371159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40731 (local address 127.0.98.62:34853)
0924 02:57:53.371203 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:53.371205 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:53.371398 (+ 193us) server_negotiation.cc:407] Connection header received
0924 02:57:53.372418 (+ 1020us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:53.372419 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:53.372429 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:53.372471 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:53.373058 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.373548 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.374552 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.374745 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.374793 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:53.376694 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:53.376696 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:53.376696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:53.376702 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:53.377729 (+ 1027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:53.378051 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:53.378052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:53.378052 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:53.378097 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:53.378261 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:53.378262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:53.378262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:53.378353 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:53.378397 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:53.378656 (+ 259us) server_negotiation.cc:299] Negotiation successful
0924 02:57:53.378712 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 02:57:53.378948 6410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:53.371134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34853 (local address 127.0.98.3:40731)
0924 02:57:53.371278 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:53.371318 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:53.372335 (+ 1017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:53.372540 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:53.372544 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:53.372807 (+ 263us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:53.372989 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:53.372994 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.373641 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.373642 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:53.374460 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:53.374462 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.374802 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.374803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:53.374905 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:53.375348 (+ 443us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:53.375356 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:53.376617 (+ 1261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:53.377790 (+ 1173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:53.377795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:53.377797 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:53.377998 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:53.378139 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:53.378139 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:53.378139 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:53.378221 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:53.378395 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:53.378398 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:53.378512 (+ 114us) client_negotiation.cc:769] Sending connection context
0924 02:57:53.378634 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 02:57:53.378750 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":68,"threads_started":1}
I20260924 02:57:53.379323 6408 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34853
I20260924 02:57:53.379412 6408 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:53.379571 6408 heartbeater.cc:511] Master 127.0.98.62:34853 requested a full tablet report, sending...
I20260924 02:57:53.379892 5949 ts_manager.cc:194] Registered new tserver with Master: 3ce8abd350504f53a0a490a9b491f09f (127.0.98.3:42231)
I20260924 02:57:53.380254 5949 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40731
I20260924 02:57:53.383391 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/test-admin.keytab.
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[5904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:53.398434 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260924 02:57:53.400002 6415 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:57:53.399271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46198 (local address 127.0.98.1:46467)
0924 02:57:53.399433 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:53.399437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:53.399449 (+ 12us) server_negotiation.cc:407] Connection header received
0924 02:57:53.399473 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:53.399475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:53.399488 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:53.399552 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:53.399932 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46198: BlockingRecv error: recv got EOF from 127.0.0.1:46198 (error 108)
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":90,"threads_started":1}
W20260924 02:57:53.400168 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[5904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:53.401106 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[5904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:53.401904 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[5904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 02:57:53.409106 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:53.402377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46202 (local address 127.0.98.1:46467)
0924 02:57:53.402429 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:53.402431 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:53.402492 (+ 61us) server_negotiation.cc:407] Connection header received
0924 02:57:53.402564 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:53.402565 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:53.402577 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:53.402625 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:53.402971 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.403776 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.404442 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.404751 (+ 309us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.404800 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:53.406807 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:53.406820 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:53.406820 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:53.406827 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:53.408204 (+ 1377us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:53.408506 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:53.408509 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:53.408509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:53.408555 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:53.408677 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:53.408677 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:53.408677 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:53.408786 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:53.408898 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:53.408951 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 02:57:53.409008 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 02:57:53.410727 6415 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:57:53.410260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46214 (local address 127.0.98.1:46467)
0924 02:57:53.410294 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:53.410295 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:53.410332 (+ 37us) server_negotiation.cc:407] Connection header received
0924 02:57:53.410374 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:53.410374 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:53.410383 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:53.410423 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:53.410686 (+ 263us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46214: BlockingRecv error: recv got EOF from 127.0.0.1:46214 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 02:57:53.410841 392 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/krb5cc' of type FILE
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[5904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:53.411680 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[5904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:53.412451 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[5904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 02:57:53.418637 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:53.412907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46216 (local address 127.0.98.1:46467)
0924 02:57:53.412956 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:53.412958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:53.413008 (+ 50us) server_negotiation.cc:407] Connection header received
0924 02:57:53.413063 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:53.413064 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:53.413074 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:53.413118 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:53.413460 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.413930 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.414609 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.414805 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.414861 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:53.416561 (+ 1700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:53.416562 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:53.416563 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:53.416569 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:53.417722 (+ 1153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:53.417995 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:53.417996 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:53.417997 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:53.418041 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:53.418194 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:53.418195 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:53.418195 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:53.418282 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:53.418359 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:53.418461 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 02:57:53.418555 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260924 02:57:53.419924 6415 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:57:53.419511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46224 (local address 127.0.98.1:46467)
0924 02:57:53.419547 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:53.419548 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:53.419577 (+ 29us) server_negotiation.cc:407] Connection header received
0924 02:57:53.419625 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:53.419626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:53.419633 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:53.419672 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:53.419881 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46224: BlockingRecv error: recv got EOF from 127.0.0.1:46224 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 02:57:53.420053 392 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790218647334776-392-0/krb5kdc/krb5cc)
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[5904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:53.420883 392 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[5904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:53.421598 392 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[5904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218673, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 02:57:53.427524 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:53.422007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46232 (local address 127.0.98.1:46467)
0924 02:57:53.422058 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:53.422059 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:53.422067 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:57:53.422121 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:53.422121 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:53.422129 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:53.422171 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:53.422485 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.422961 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.423590 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:53.423763 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:53.423813 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:53.425610 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:53.425611 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:53.425611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:53.425618 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:53.426645 (+ 1027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:53.426929 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:53.426930 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:53.426930 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:53.426974 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:53.427117 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:53.427118 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:53.427118 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:53.427200 (+ 82us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:53.427274 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:53.427378 (+ 104us) server_negotiation.cc:299] Negotiation successful
0924 02:57:53.427441 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 02:57:53.428081 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 6011
I20260924 02:57:53.432448 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 6145
I20260924 02:57:53.436002 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 6279
I20260924 02:57:53.439487 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 5917
2026-09-24T02:57:53Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1742 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[6428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[6428](info): set up 2 sockets
Sep 24 02:57:53 dist-test-slave-sb6p krb5kdc[6428](info): commencing operation
krb5kdc: starting...
W20260924 02:57:54.386359 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:57:54 dist-test-slave-sb6p krb5kdc[6428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:57:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:57:54Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 02:57:54.495302 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37993
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37993
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:54.572736 6441 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:54.572917 6441 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:54.572938 6441 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:54.574043 6441 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:54.574083 6441 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:54.574095 6441 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:54.574107 6441 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:54.574131 6441 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:57:54.575580 6441 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37993
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6441
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:54.575798 6441 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:54.575996 6441 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:54.580471 6447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:54.580466 6449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:54.580466 6446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:54.580554 6441 server_base.cc:1034] running on GCE node
I20260924 02:57:54.580897 6441 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:54.581152 6441 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:54.582281 6441 hybrid_clock.cc:648] HybridClock initialized: now 1790218674582266 us; error 31 us; skew 500 ppm
Sep 24 02:57:54 dist-test-slave-sb6p krb5kdc[6428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218674, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:54.584520 6441 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 02:57:54.584933 6441 webserver.cc:460] Webserver started at http://127.0.98.62:44925/ using document root <none> and password file <none>
I20260924 02:57:54.585137 6441 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:54.585183 6441 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:54.585301 6441 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:54.586297 6441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "eac45a5647564b07822c25ba28440268"
format_stamp: "Formatted at 2026-09-24 02:57:54 on dist-test-slave-sb6p"
server_key: "dd925b232d26b46cb010b5079efca7ea"
server_key_iv: "e0d5700049700fc7292e45eb604b30fe"
server_key_version: "encryptionkey@0"
I20260924 02:57:54.586637 6441 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "eac45a5647564b07822c25ba28440268"
format_stamp: "Formatted at 2026-09-24 02:57:54 on dist-test-slave-sb6p"
server_key: "dd925b232d26b46cb010b5079efca7ea"
server_key_iv: "e0d5700049700fc7292e45eb604b30fe"
server_key_version: "encryptionkey@0"
I20260924 02:57:54.587821 6441 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 02:57:54.588526 6456 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:54.588672 6441 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:54.588749 6441 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "eac45a5647564b07822c25ba28440268"
format_stamp: "Formatted at 2026-09-24 02:57:54 on dist-test-slave-sb6p"
server_key: "dd925b232d26b46cb010b5079efca7ea"
server_key_iv: "e0d5700049700fc7292e45eb604b30fe"
server_key_version: "encryptionkey@0"
I20260924 02:57:54.588811 6441 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:54.606046 6441 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:54.607715 6441 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:54.607852 6441 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:54.611500 6441 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37993
I20260924 02:57:54.611541 6508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37993 every 8 connection(s)
I20260924 02:57:54.611850 6441 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:57:54.612746 6509 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:54.614943 6509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268: Bootstrap starting.
I20260924 02:57:54.615489 6509 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:54.615741 6509 log.cc:824] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:54.616340 6509 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268: No bootstrap required, opened a new log
I20260924 02:57:54.617859 6509 raft_consensus.cc:348] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260924 02:57:54.617996 6509 raft_consensus.cc:374] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:54.618028 6509 raft_consensus.cc:729] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eac45a5647564b07822c25ba28440268, State: Initialized, Role: FOLLOWER
I20260924 02:57:54.618166 6509 consensus_queue.cc:260] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260924 02:57:54.618225 6509 raft_consensus.cc:388] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:57:54.618247 6509 raft_consensus.cc:482] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:57:54.618291 6509 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:54.619112 6509 raft_consensus.cc:504] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260924 02:57:54.619200 6509 leader_election.cc:302] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eac45a5647564b07822c25ba28440268; no voters:
I20260924 02:57:54.619365 6509 leader_election.cc:288] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:57:54.619395 6512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:54.619446 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 6441
I20260924 02:57:54.619491 6512 raft_consensus.cc:686] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [term 1 LEADER]: Becoming Leader. State: Replica: eac45a5647564b07822c25ba28440268, State: Running, Role: LEADER
I20260924 02:57:54.619540 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:57:54.619560 6512 consensus_queue.cc:237] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } }
I20260924 02:57:54.619622 6509 sys_catalog.cc:564] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:54.619693 392 external_mini_cluster.cc:1428] Setting key f7b87109070c9e469a3a9f2db4d68dc0
I20260924 02:57:54.619838 6512 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } }
I20260924 02:57:54.619905 6512 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:54.620011 6512 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eac45a5647564b07822c25ba28440268. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } }
I20260924 02:57:54.620051 6512 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:54.620687 6516 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:54.621194 6516 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 02:57:54 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 02:57:54.626370 6516 catalog_manager.cc:1347] Generated new cluster ID: 58c36acd554e4645bd70b6b2991339b9
I20260924 02:57:54.626412 6516 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:57:54.630491 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:54.620248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34568 (local address 127.0.98.62:37993)
0924 02:57:54.620500 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:54.620510 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:54.620542 (+ 32us) server_negotiation.cc:407] Connection header received
0924 02:57:54.620707 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:54.620714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:54.620788 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:54.620952 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:54.621499 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:54.622839 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:54.623541 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:54.624147 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:54.624255 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:54.626570 (+ 2315us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:54.626574 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:54.626575 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:54.626586 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:54.628975 (+ 2389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:54.629353 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:54.629356 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:54.629356 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:54.629451 (+ 95us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:54.629627 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:54.629627 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:54.629628 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:54.629917 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:54.630052 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:54.630147 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 02:57:54.630279 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206,"thread_start_us":100,"threads_started":1}
I20260924 02:57:54.649436 6516 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:57:54.650009 6516 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:54.666601 6516 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268: Generated new TSK 0
I20260924 02:57:54.666796 6516 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 02:57:54.694418 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:54.782471 6535 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:54.782696 6535 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:54.782725 6535 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:54.783855 6535 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:54.783895 6535 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:54.783929 6535 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:57:54.785324 6535 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6535
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:54.785555 6535 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:54.785754 6535 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:54.790616 6540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:54.790696 6535 server_base.cc:1034] running on GCE node
W20260924 02:57:54.790702 6541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:54.790622 6543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:54.790947 6535 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:54.791162 6535 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:54.792320 6535 hybrid_clock.cc:648] HybridClock initialized: now 1790218674792304 us; error 32 us; skew 500 ppm
Sep 24 02:57:54 dist-test-slave-sb6p krb5kdc[6428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218674, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:54.794723 6535 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 02:57:54.795135 6535 webserver.cc:460] Webserver started at http://127.0.98.1:35971/ using document root <none> and password file <none>
I20260924 02:57:54.795342 6535 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:54.795388 6535 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:54.795507 6535 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:54.796465 6535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "6cda0e259079447197b80ec4eb9acf9f"
format_stamp: "Formatted at 2026-09-24 02:57:54 on dist-test-slave-sb6p"
server_key: "e8eedcd57a90e08a205b1dbe26e6547e"
server_key_iv: "ca1f65a1f7760d6a1caf48d9db1424d5"
server_key_version: "encryptionkey@0"
I20260924 02:57:54.796778 6535 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "6cda0e259079447197b80ec4eb9acf9f"
format_stamp: "Formatted at 2026-09-24 02:57:54 on dist-test-slave-sb6p"
server_key: "e8eedcd57a90e08a205b1dbe26e6547e"
server_key_iv: "ca1f65a1f7760d6a1caf48d9db1424d5"
server_key_version: "encryptionkey@0"
I20260924 02:57:54.797940 6535 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 02:57:54.798630 6550 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:54.798803 6535 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:54.798887 6535 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "6cda0e259079447197b80ec4eb9acf9f"
format_stamp: "Formatted at 2026-09-24 02:57:54 on dist-test-slave-sb6p"
server_key: "e8eedcd57a90e08a205b1dbe26e6547e"
server_key_iv: "ca1f65a1f7760d6a1caf48d9db1424d5"
server_key_version: "encryptionkey@0"
I20260924 02:57:54.798944 6535 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:54.819497 6535 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:54.821713 6535 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:54.821833 6535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:54.822007 6535 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:54.822284 6535 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:54.822317 6535 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:54.822338 6535 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:54.822361 6535 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:54.827845 6535 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35253
I20260924 02:57:54.827914 6663 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35253 every 8 connection(s)
I20260924 02:57:54.828210 6535 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:57:54.829160 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 6535
I20260924 02:57:54.829267 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:57:54.829459 392 external_mini_cluster.cc:1428] Setting key c2c4f6ff50bacaa00a7137940ccc7e54
Sep 24 02:57:54 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218674, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 02:57:54.837531 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:54.828884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55523 (local address 127.0.98.62:37993)
0924 02:57:54.828929 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:54.828931 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:54.829175 (+ 244us) server_negotiation.cc:407] Connection header received
0924 02:57:54.829803 (+ 628us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:54.829805 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:54.829815 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:54.829855 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:54.831032 (+ 1177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:54.831521 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:54.832558 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:54.832766 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:54.832822 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:54.835004 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:54.835005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:54.835006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:54.835012 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:54.836242 (+ 1230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:54.836674 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:54.836675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:54.836675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:54.836718 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:54.836966 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:54.836967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:54.836967 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:54.837081 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:54.837132 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:54.837403 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 02:57:54.837466 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 02:57:54.837682 6666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:54.828857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.1:55523)
0924 02:57:54.829027 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:54.829072 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:54.829710 (+ 638us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:54.829927 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:54.829931 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:54.830670 (+ 739us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:54.830936 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:54.830942 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:54.831631 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:54.831632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:54.832476 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:54.832478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:54.832838 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:54.832839 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:54.832972 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:54.833459 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:54.833468 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:54.834914 (+ 1446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:54.836318 (+ 1404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:54.836325 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:54.836328 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:54.836604 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:54.836771 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:54.836772 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:54.836772 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:54.836904 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:54.837142 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:54.837148 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:54.837296 (+ 148us) client_negotiation.cc:769] Sending connection context
0924 02:57:54.837384 (+ 88us) client_negotiation.cc:240] Negotiation successful
0924 02:57:54.837499 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260924 02:57:54.838127 6664 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37993
I20260924 02:57:54.838224 6664 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:54.838433 6664 heartbeater.cc:511] Master 127.0.98.62:37993 requested a full tablet report, sending...
I20260924 02:57:54.838969 6473 ts_manager.cc:194] Registered new tserver with Master: 6cda0e259079447197b80ec4eb9acf9f (127.0.98.1:35253)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
I20260924 02:57:54.839622 6473 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:55523
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 02:57:54.851369 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:54.928483 6669 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:54.928682 6669 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:54.928704 6669 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:54.929797 6669 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:54.929832 6669 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:54.929857 6669 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:57:54.931288 6669 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6669
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:54.931490 6669 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:54.931663 6669 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:54.936339 6677 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:54.936354 6675 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:54.936322 6674 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:54.936570 6669 server_base.cc:1034] running on GCE node
I20260924 02:57:54.936755 6669 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:54.936985 6669 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:54.938124 6669 hybrid_clock.cc:648] HybridClock initialized: now 1790218674938110 us; error 27 us; skew 500 ppm
Sep 24 02:57:54 dist-test-slave-sb6p krb5kdc[6428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218674, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:54.940460 6669 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 02:57:54.940861 6669 webserver.cc:460] Webserver started at http://127.0.98.2:33271/ using document root <none> and password file <none>
I20260924 02:57:54.941061 6669 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:54.941107 6669 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:54.941226 6669 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:54.942206 6669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "50ada8ae64f647f8826a849c1377ac53"
format_stamp: "Formatted at 2026-09-24 02:57:54 on dist-test-slave-sb6p"
server_key: "093946ac96722bf1bad8bdcd53d8c459"
server_key_iv: "1761d79f8fb3243d31df33d00c7e0fc3"
server_key_version: "encryptionkey@0"
I20260924 02:57:54.942543 6669 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "50ada8ae64f647f8826a849c1377ac53"
format_stamp: "Formatted at 2026-09-24 02:57:54 on dist-test-slave-sb6p"
server_key: "093946ac96722bf1bad8bdcd53d8c459"
server_key_iv: "1761d79f8fb3243d31df33d00c7e0fc3"
server_key_version: "encryptionkey@0"
I20260924 02:57:54.943696 6669 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 02:57:54.944315 6684 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:54.944450 6669 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:54.944526 6669 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "50ada8ae64f647f8826a849c1377ac53"
format_stamp: "Formatted at 2026-09-24 02:57:54 on dist-test-slave-sb6p"
server_key: "093946ac96722bf1bad8bdcd53d8c459"
server_key_iv: "1761d79f8fb3243d31df33d00c7e0fc3"
server_key_version: "encryptionkey@0"
I20260924 02:57:54.944589 6669 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:54.962841 6669 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:54.964501 6669 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:54.964634 6669 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:54.964855 6669 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:54.965186 6669 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:54.965220 6669 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:54.965250 6669 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:54.965267 6669 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:54.970547 6669 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45269
I20260924 02:57:54.970578 6797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45269 every 8 connection(s)
I20260924 02:57:54.970880 6669 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 02:57:54.975227 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 6669
I20260924 02:57:54.975330 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:57:54.975507 392 external_mini_cluster.cc:1428] Setting key 23136c86bc5801db90f297e779f2ee73
Sep 24 02:57:54 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218674, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 02:57:54.979660 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:54.971489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54193 (local address 127.0.98.62:37993)
0924 02:57:54.971533 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:54.971535 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:54.971743 (+ 208us) server_negotiation.cc:407] Connection header received
0924 02:57:54.972222 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:54.972223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:54.972235 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:54.972287 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:54.973206 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:54.973687 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:54.974695 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:54.974891 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:54.974946 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:54.977144 (+ 2198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:54.977146 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:54.977146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:54.977154 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:54.978380 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:54.978784 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:54.978785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:54.978786 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:54.978840 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:54.979068 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:54.979069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:54.979069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:54.979176 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:54.979232 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:54.979490 (+ 258us) server_negotiation.cc:299] Negotiation successful
0924 02:57:54.979573 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 02:57:54.979862 6800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:54.971474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.2:54193)
0924 02:57:54.971628 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:54.971662 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:54.972128 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:54.972382 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:54.972388 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:54.972920 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:54.973106 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:54.973110 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:54.973769 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:54.973770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:54.974620 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:54.974623 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:54.974957 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:54.974958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:54.975122 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:54.975628 (+ 506us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:54.975636 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:54.977058 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:54.978455 (+ 1397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:54.978462 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:54.978464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:54.978720 (+ 256us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:54.978902 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:54.978902 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:54.978903 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:54.979013 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:54.979229 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:54.979234 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:54.979376 (+ 142us) client_negotiation.cc:769] Sending connection context
0924 02:57:54.979462 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 02:57:54.979615 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":79,"threads_started":1}
I20260924 02:57:54.980284 6798 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37993
I20260924 02:57:54.980376 6798 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:54.980536 6798 heartbeater.cc:511] Master 127.0.98.62:37993 requested a full tablet report, sending...
I20260924 02:57:54.980918 6473 ts_manager.cc:194] Registered new tserver with Master: 50ada8ae64f647f8826a849c1377ac53 (127.0.98.2:45269)
I20260924 02:57:54.981269 6473 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:54193
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 02:57:54.997303 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:55.074416 6803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:55.074666 6803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:55.074697 6803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:55.075860 6803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:55.075901 6803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:55.075929 6803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:57:55.077363 6803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37993
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:55.077575 6803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:55.077778 6803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:55.082638 6808 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:55.082638 6809 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:55.082789 6811 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:55.083427 6803 server_base.cc:1034] running on GCE node
I20260924 02:57:55.083628 6803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:55.083900 6803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:55.085045 6803 hybrid_clock.cc:648] HybridClock initialized: now 1790218675085020 us; error 47 us; skew 500 ppm
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:55.087517 6803 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 02:57:55.087883 6803 webserver.cc:460] Webserver started at http://127.0.98.3:45695/ using document root <none> and password file <none>
I20260924 02:57:55.088048 6803 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:55.088085 6803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:55.088172 6803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:55.089226 6803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "1631c5c9efc94c31975d48812ba9f320"
format_stamp: "Formatted at 2026-09-24 02:57:55 on dist-test-slave-sb6p"
server_key: "4526473c018e15d9b9a1635734ce862a"
server_key_iv: "51c933242f14a759228110c755436dd4"
server_key_version: "encryptionkey@0"
I20260924 02:57:55.089519 6803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1631c5c9efc94c31975d48812ba9f320"
format_stamp: "Formatted at 2026-09-24 02:57:55 on dist-test-slave-sb6p"
server_key: "4526473c018e15d9b9a1635734ce862a"
server_key_iv: "51c933242f14a759228110c755436dd4"
server_key_version: "encryptionkey@0"
I20260924 02:57:55.090718 6803 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 02:57:55.091328 6818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.091495 6803 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.091573 6803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "1631c5c9efc94c31975d48812ba9f320"
format_stamp: "Formatted at 2026-09-24 02:57:55 on dist-test-slave-sb6p"
server_key: "4526473c018e15d9b9a1635734ce862a"
server_key_iv: "51c933242f14a759228110c755436dd4"
server_key_version: "encryptionkey@0"
I20260924 02:57:55.091627 6803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:55.105007 6803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:55.107023 6803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:55.107131 6803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:55.107311 6803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:55.107590 6803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:55.107622 6803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.107651 6803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:55.107679 6803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.112934 6803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43571
I20260924 02:57:55.112995 6931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43571 every 8 connection(s)
I20260924 02:57:55.113266 6803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 02:57:55.121130 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 6803
I20260924 02:57:55.121238 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:57:55.121449 392 external_mini_cluster.cc:1428] Setting key 6f0c6d162ba43ff3938b497d1ee4ac00
I20260924 02:57:55.121981 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.113928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37793 (local address 127.0.98.62:37993)
0924 02:57:55.113971 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.113973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.114171 (+ 198us) server_negotiation.cc:407] Connection header received
0924 02:57:55.114678 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.114680 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.114691 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.114738 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.115658 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.116137 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.117109 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.117290 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.117342 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.119248 (+ 1906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.119249 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.119250 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.119257 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.120650 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.121026 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.121028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.121028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.121094 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.121345 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.121346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.121346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.121456 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.121526 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.121801 (+ 275us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.121886 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 02:57:55.122154 6934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.113899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.3:37793)
0924 02:57:55.114054 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.114089 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.114575 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.114818 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.114823 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.115369 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.115560 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.115564 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.116222 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.116223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.117027 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.117030 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.117353 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.117354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.117475 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.117884 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.117892 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.119163 (+ 1271us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.120724 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.120729 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.120731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.120948 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.121157 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.121158 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.121158 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.121290 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.121515 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.121520 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.121660 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.121769 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.121930 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":75,"threads_started":1}
I20260924 02:57:55.122583 6932 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37993
I20260924 02:57:55.122671 6932 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:55.122834 6932 heartbeater.cc:511] Master 127.0.98.62:37993 requested a full tablet report, sending...
I20260924 02:57:55.123174 6473 ts_manager.cc:194] Registered new tserver with Master: 1631c5c9efc94c31975d48812ba9f320 (127.0.98.3:43571)
I20260924 02:57:55.123517 6473 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37793
I20260924 02:57:55.133105 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.61.keytab.
Entry for principal oryx/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.61.keytab.
I20260924 02:57:55.154769 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41113
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37993,127.0.98.61:41113
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:55.231940 6937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:55.232137 6937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:55.232167 6937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:55.233275 6937 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:55.233316 6937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:55.233335 6937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:55.233366 6937 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:55.233382 6937 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:57:55.234822 6937 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37993,127.0.98.61:41113
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.61.keytab
--principal=oryx/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:41113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
W20260924 02:57:55.234920 6937 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.98.62:37993,127.0.98.61:41113'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260924 02:57:55.235069 6937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:55.235263 6937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:55.239611 6942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:55.239619 6943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:55.239624 6945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:55.239815 6937 server_base.cc:1034] running on GCE node
I20260924 02:57:55.240024 6937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:55.240273 6937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:55.241417 6937 hybrid_clock.cc:648] HybridClock initialized: now 1790218675241412 us; error 32 us; skew 500 ppm
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:55.243662 6937 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 02:57:55.244033 6937 webserver.cc:460] Webserver started at http://127.0.98.61:37819/ using document root <none> and password file <none>
I20260924 02:57:55.244228 6937 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:55.244273 6937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:55.244396 6937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:55.245350 6937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/data/instance:
uuid: "a8332978b13547c88de3e29f5b370cf3"
format_stamp: "Formatted at 2026-09-24 02:57:55 on dist-test-slave-sb6p"
server_key: "d1b7b6220765eeb6060242dd5a6aeba9"
server_key_iv: "4949ba8722ea8db7a4b06f7c7b813aeb"
server_key_version: "encryptionkey@0"
I20260924 02:57:55.245656 6937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/wal/instance:
uuid: "a8332978b13547c88de3e29f5b370cf3"
format_stamp: "Formatted at 2026-09-24 02:57:55 on dist-test-slave-sb6p"
server_key: "d1b7b6220765eeb6060242dd5a6aeba9"
server_key_iv: "4949ba8722ea8db7a4b06f7c7b813aeb"
server_key_version: "encryptionkey@0"
I20260924 02:57:55.246865 6937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 02:57:55.247524 6952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.247661 6937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:55.247736 6937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/wal
uuid: "a8332978b13547c88de3e29f5b370cf3"
format_stamp: "Formatted at 2026-09-24 02:57:55 on dist-test-slave-sb6p"
server_key: "d1b7b6220765eeb6060242dd5a6aeba9"
server_key_iv: "4949ba8722ea8db7a4b06f7c7b813aeb"
server_key_version: "encryptionkey@0"
I20260924 02:57:55.247793 6937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:55.258255 6937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:55.259876 6937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:55.259999 6937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:55.263515 6937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41113
I20260924 02:57:55.263547 7004 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41113 every 8 connection(s)
I20260924 02:57:55.263842 6937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/data/info.pb
I20260924 02:57:55.264717 7005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:55.266121 7005 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } has no permanent_uuid. Determining permanent_uuid...
I20260924 02:57:55.268610 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 6937
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 02:57:55.274912 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.266812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34570 (local address 127.0.98.62:37993)
0924 02:57:55.266861 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.266863 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.267070 (+ 207us) server_negotiation.cc:407] Connection header received
0924 02:57:55.267497 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.267498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.267508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.267558 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.268114 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.268687 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.269930 (+ 1243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.270129 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.270178 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.272352 (+ 2174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.272355 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.272355 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.272362 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.273539 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.273888 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.273889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.273889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.273937 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.274196 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.274197 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.274197 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.274372 (+ 175us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.274442 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.274732 (+ 290us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.274809 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 02:57:55.275091 7007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.266793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.0.1:34570)
0924 02:57:55.266958 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.266993 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.267424 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.267630 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.267633 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.267864 (+ 231us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.268052 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.268056 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.268784 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.268785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.269848 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.269851 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.270195 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.270197 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.270331 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.270789 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.270798 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.272265 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.273618 (+ 1353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.273625 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.273626 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.273828 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.274002 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.274002 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.274003 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.274136 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.274437 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.274441 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.274598 (+ 157us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.274707 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.274885 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":80,"threads_started":1}
I20260924 02:57:55.275689 7005 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } has no permanent_uuid. Determining permanent_uuid...
I20260924 02:57:55.277030 7009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.269076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38754 (local address 127.0.98.61:41113)
0924 02:57:55.269233 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.269237 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.269259 (+ 22us) server_negotiation.cc:407] Connection header received
0924 02:57:55.269299 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.269301 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.269319 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.269402 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.269919 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.270805 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.271571 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.271847 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.271912 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.274284 (+ 2372us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.274291 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.274292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.274297 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.276003 (+ 1706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.276365 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.276368 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.276368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.276425 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.276606 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.276606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.276607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.276721 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.276844 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.276912 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.276965 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 02:57:55.281836 7007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.275920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41113 (local address 127.0.0.1:38756)
0924 02:57:55.275952 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.275961 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.275992 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.276199 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.276200 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.276291 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.276346 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.276348 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.277108 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.277109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.277763 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.277765 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.278058 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.278058 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.278114 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.278429 (+ 315us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.278434 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.279519 (+ 1085us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.281159 (+ 1640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.281160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.281160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.281298 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.281439 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.281439 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.281440 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.281480 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.281627 (+ 147us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.281627 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.281657 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.281699 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.281762 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 02:57:55.281934 7010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.275971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38756 (local address 127.0.98.61:41113)
0924 02:57:55.276099 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.276101 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.276106 (+ 5us) server_negotiation.cc:407] Connection header received
0924 02:57:55.276120 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.276120 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.276129 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.276190 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.276411 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.277032 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.277826 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.278011 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.278050 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.279587 (+ 1537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.279588 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.279588 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.279593 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.281099 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.281357 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.281358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.281358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.281400 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.281512 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.281512 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.281512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.281590 (+ 78us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.281629 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.281714 (+ 85us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.281803 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":112,"thread_start_us":83,"threads_started":1}
I20260924 02:57:55.283093 7005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: Bootstrap starting.
I20260924 02:57:55.283641 7005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:55.283845 7005 log.cc:824] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:55.284467 7005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: No bootstrap required, opened a new log
I20260924 02:57:55.285744 7005 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } }
I20260924 02:57:55.285859 7005 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:55.285887 7005 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8332978b13547c88de3e29f5b370cf3, State: Initialized, Role: FOLLOWER
I20260924 02:57:55.285984 7005 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } }
I20260924 02:57:55.286254 7005 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:55.286214 7012 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } } }
I20260924 02:57:55.286304 7012 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 02:57:55.287724 7023 catalog_manager.cc:1558] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 02:57:55.287776 7023 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 02:57:55.289140 6937 master_runner.cc:428] Detected that this master a8332978b13547c88de3e29f5b370cf3 is joining an existing cluster
I20260924 02:57:55.289176 6937 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.61:41113
I20260924 02:57:55.296419 7030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.290125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.0.1:34574)
0924 02:57:55.290234 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.290243 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.290294 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.290474 (+ 180us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.290475 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.290789 (+ 314us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.290872 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.290874 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.291794 (+ 920us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.291810 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.292442 (+ 632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.292445 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.292780 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.292781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.292836 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.293189 (+ 353us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.293194 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.293542 (+ 348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.295394 (+ 1852us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.295395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.295395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.295615 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.295823 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.295824 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.295824 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.295882 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.296128 (+ 246us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.296129 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.296171 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.296239 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.296320 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":72,"threads_started":1}
I20260924 02:57:55.296445 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.290048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34574 (local address 127.0.98.62:37993)
0924 02:57:55.290088 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.290090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.290312 (+ 222us) server_negotiation.cc:407] Connection header received
0924 02:57:55.290384 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.290385 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.290397 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.290456 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.290940 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.291718 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.292511 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.292710 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.292761 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.293605 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.293607 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.293607 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.293614 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.295327 (+ 1713us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.295687 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.295688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.295689 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.295752 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.295943 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.295944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.295944 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.296072 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.296140 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.296272 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.296346 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 02:57:55.297041 7029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.290043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41113 (local address 127.0.0.1:38770)
0924 02:57:55.290178 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.290187 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.290257 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.290604 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.290605 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.290842 (+ 237us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.290899 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.290901 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.291770 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.291771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.292667 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.292669 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.293157 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.293157 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.293210 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.293548 (+ 338us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.293553 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.293886 (+ 333us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.296085 (+ 2199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.296086 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.296086 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.296272 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.296446 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.296446 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.296446 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.296490 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.296724 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.296724 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.296768 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.296834 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.296926 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":32,"thread_start_us":92,"threads_started":1}
I20260924 02:57:55.297108 7010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.290044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38770 (local address 127.0.98.61:41113)
0924 02:57:55.290077 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.290079 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.290235 (+ 156us) server_negotiation.cc:407] Connection header received
0924 02:57:55.290499 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.290500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.290509 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.290575 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.290985 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.291703 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.292755 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.293056 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.293095 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.293965 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.293967 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.293967 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.293973 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.296034 (+ 2061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.296348 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.296349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.296349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.296391 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.296544 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.296544 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.296545 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.296651 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.296687 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.296867 (+ 180us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.296987 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":17}
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218674, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 02:57:55.304687 7010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.298555 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38786 (local address 127.0.98.61:41113)
0924 02:57:55.298583 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.298585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.298736 (+ 151us) server_negotiation.cc:407] Connection header received
0924 02:57:55.298803 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.298804 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.298812 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.298851 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.299227 (+ 376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.299654 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.300364 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.300552 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.300602 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.302627 (+ 2025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.302628 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.302628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.302633 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.303700 (+ 1067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.304089 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.304090 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.304090 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.304128 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.304290 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.304290 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.304291 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.304382 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.304423 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.304585 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.304633 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 02:57:55.304785 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.298491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41113 (local address 127.0.0.1:38786)
0924 02:57:55.298651 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.298682 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.298745 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.298872 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.298875 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.299118 (+ 243us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.299160 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.299162 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.299716 (+ 554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.299717 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.300287 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.300290 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.300608 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.300609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.300693 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.301087 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.301091 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.302559 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.303767 (+ 1208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.303773 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.303775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.304024 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.304178 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.304178 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.304179 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.304242 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.304430 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.304435 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.304487 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.304571 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.304680 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":82,"threads_started":1}
I20260924 02:57:55.305294 6473 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.61:41113: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "a8332978b13547c88de3e29f5b370cf3"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.61"
port: 41113
}
attrs {
promote: true
}
}
dest_uuid: "eac45a5647564b07822c25ba28440268"
cas_config_opid_index: -1
I20260924 02:57:55.305574 6473 consensus_queue.cc:237] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } }
I20260924 02:57:55.305994 7032 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a8332978b13547c88de3e29f5b370cf3 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } } }
I20260924 02:57:55.306259 6979 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:55.307016 7032 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.61:41113
I20260924 02:57:55.307055 6979 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 1 FOLLOWER]: Refusing update from remote peer eac45a5647564b07822c25ba28440268: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260924 02:57:55.307215 6937 master.cc:537] Master@127.0.98.61:41113 shutting down...
I20260924 02:57:55.307456 7034 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 02:57:55.308200 6459 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260924 02:57:55.308877 7033 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } } }
I20260924 02:57:55.309315 7033 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:55.309952 6937 raft_consensus.cc:2229] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:55.310029 6937 raft_consensus.cc:2258] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:55.310089 6937 tablet_replica.cc:330] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: stopping tablet replica
I20260924 02:57:55.314878 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.314761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41113 (local address 127.0.0.1:38802)
0924 02:57:55.314810 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.314851 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41113: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 02:57:55.314968 6459 consensus_peers.cc:487] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 -> Peer a8332978b13547c88de3e29f5b370cf3 (127.0.98.61:41113): Couldn't send request to peer a8332978b13547c88de3e29f5b370cf3. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.61:41113: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 02:57:55.322888 6937 master.cc:552] Master@127.0.98.61:41113 shutdown complete.
I20260924 02:57:55.322942 6937 master_runner.cc:305] Clearing existing system tablet
I20260924 02:57:55.323060 6937 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:55.323103 6937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:55.323781 6937 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:55.324244 7038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.324388 6937 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:55.324440 6937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.324462 6937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/wal
uuid: "a8332978b13547c88de3e29f5b370cf3"
format_stamp: "Formatted at 2026-09-24 02:57:55 on dist-test-slave-sb6p"
server_key: "d1b7b6220765eeb6060242dd5a6aeba9"
server_key_iv: "4949ba8722ea8db7a4b06f7c7b813aeb"
server_key_version: "encryptionkey@0"
I20260924 02:57:55.325047 6937 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:55.326295 6937 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 02:57:55.326359 6937 log.cc:1196] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260924 02:57:55.326537 6937 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: Deleting consensus metadata
I20260924 02:57:55.326645 6937 master_runner.cc:315] Copying system tablet from 127.0.98.62:37993
I20260924 02:57:55.327191 6937 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37993
I20260924 02:57:55.332947 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.327534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34582 (local address 127.0.98.62:37993)
0924 02:57:55.327582 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.327584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.327624 (+ 40us) server_negotiation.cc:407] Connection header received
0924 02:57:55.327701 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.327702 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.327710 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.327754 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.328124 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.328601 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.329640 (+ 1039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.329815 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.329868 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.330822 (+ 954us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.330823 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.330824 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.330830 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.332007 (+ 1177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.332357 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.332357 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.332357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.332400 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.332562 (+ 162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.332562 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.332562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.332650 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.332694 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.332825 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.332886 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 02:57:55.333118 7044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.327460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.0.1:34582)
0924 02:57:55.327559 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.327571 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.327642 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.327760 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.327761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.327961 (+ 200us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.328024 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.328026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.328670 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.328671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.329569 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.329571 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.329882 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.329883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.329944 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.330339 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.330343 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.330750 (+ 407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.332081 (+ 1331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.332082 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.332083 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.332293 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.332450 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.332450 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.332451 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.332513 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.332698 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.332698 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.332742 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.332810 (+ 68us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.332906 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":58,"threads_started":1}
I20260924 02:57:55.333528 6493 tablet_copy_service.cc:140] P eac45a5647564b07822c25ba28440268: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer a8332978b13547c88de3e29f5b370cf3 ({username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34582)
I20260924 02:57:55.333631 6493 tablet_copy_service.cc:161] P eac45a5647564b07822c25ba28440268: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer a8332978b13547c88de3e29f5b370cf3 at {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34582: session id = a8332978b13547c88de3e29f5b370cf3-00000000000000000000000000000000
I20260924 02:57:55.334435 6493 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 02:57:55.335033 6937 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 02:57:55.335896 6937 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: tablet copy: Starting download of 0 data blocks...
I20260924 02:57:55.336026 6937 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: tablet copy: Starting download of 1 WAL segments...
I20260924 02:57:55.336833 6937 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 02:57:55.337538 6493 tablet_copy_service.cc:342] P eac45a5647564b07822c25ba28440268: Request end of tablet copy session a8332978b13547c88de3e29f5b370cf3-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.61@KRBTEST.COM'} at 127.0.0.1:34582
I20260924 02:57:55.337597 6493 tablet_copy_service.cc:433] P eac45a5647564b07822c25ba28440268: ending tablet copy session a8332978b13547c88de3e29f5b370cf3-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer a8332978b13547c88de3e29f5b370cf3
I20260924 02:57:55.340094 6937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:55.341125 7045 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:55.341239 7046 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:55.341280 7048 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:55.341351 6937 server_base.cc:1034] running on GCE node
I20260924 02:57:55.341536 6937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:55.341657 6937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:55.342752 6937 hybrid_clock.cc:648] HybridClock initialized: now 1790218675342746 us; error 22 us; skew 500 ppm
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:55.344020 6937 init.cc:377] Logged in from keytab as oryx/127.0.98.61@KRBTEST.COM (short username oryx)
I20260924 02:57:55.344254 6937 webserver.cc:460] Webserver started at http://127.0.98.61:39347/ using document root <none> and password file <none>
I20260924 02:57:55.344385 6937 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:55.344419 6937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:55.345026 6937 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:55.345520 7055 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.345618 6937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.345659 6937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/wal
uuid: "a8332978b13547c88de3e29f5b370cf3"
format_stamp: "Formatted at 2026-09-24 02:57:55 on dist-test-slave-sb6p"
server_key: "d1b7b6220765eeb6060242dd5a6aeba9"
server_key_iv: "4949ba8722ea8db7a4b06f7c7b813aeb"
server_key_version: "encryptionkey@0"
I20260924 02:57:55.345705 6937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:55.366113 6937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:55.367643 6937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:55.369958 6937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:41113
I20260924 02:57:55.370052 7107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:41113 every 8 connection(s)
I20260924 02:57:55.370230 6937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-1/data/info.pb
I20260924 02:57:55.370951 7108 sys_catalog.cc:263] Verifying existing consensus state
I20260924 02:57:55.371317 7108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: Bootstrap starting.
I20260924 02:57:55.373355 7108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 02:57:55.373538 7108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: Bootstrap complete.
I20260924 02:57:55.373719 7108 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } }
I20260924 02:57:55.373785 7108 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: a8332978b13547c88de3e29f5b370cf3, State: Initialized, Role: LEARNER
I20260924 02:57:55.373808 7108 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } }
I20260924 02:57:55.373979 7108 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:55.373984 7111 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } } }
I20260924 02:57:55.374112 7111 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: This master's current role is: LEARNER
I20260924 02:57:55.375490 7122 catalog_manager.cc:1259] Loaded cluster ID: 58c36acd554e4645bd70b6b2991339b9
I20260924 02:57:55.375541 7122 catalog_manager.cc:1552] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: loading cluster ID for follower catalog manager: success
I20260924 02:57:55.376647 7122 catalog_manager.cc:1574] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: acquiring CA information for follower catalog manager: success
I20260924 02:57:55.376964 7122 catalog_manager.cc:1602] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 02:57:55.384007 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.378404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38820 (local address 127.0.98.61:41113)
0924 02:57:55.378538 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.378540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.378548 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:57:55.378569 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.378570 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.378579 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.378624 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.379124 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.379744 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.380429 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.380623 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.380674 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.381758 (+ 1084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.381760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.381760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.381767 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.383059 (+ 1292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.383359 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.383360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.383360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.383408 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.383564 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.383565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.383565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.383659 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.383740 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.383845 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.383921 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
WARNING: no policy specified for oryx/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.60.keytab.
Entry for principal oryx/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.60.keytab.
I20260924 02:57:55.406566 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36985
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37993,127.0.98.61:41113,127.0.98.60:36985
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:55.486281 7126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:55.486639 7126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:55.486696 7126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:55.487895 7126 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:55.487933 7126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:55.487946 7126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:55.487957 7126 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:55.487974 7126 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:57:55.489468 7126 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37993,127.0.98.61:41113,127.0.98.60:36985
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.60.keytab
--principal=oryx/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:36985
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:55.489655 7126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:55.489823 7126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:55.494329 7132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:55.494414 7131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:55.494331 7134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:55.494558 7126 server_base.cc:1034] running on GCE node
I20260924 02:57:55.494853 7126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:55.495083 7126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:55.496207 7126 hybrid_clock.cc:648] HybridClock initialized: now 1790218675496191 us; error 32 us; skew 500 ppm
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:55.498412 7126 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 02:57:55.498862 7126 webserver.cc:460] Webserver started at http://127.0.98.60:34479/ using document root <none> and password file <none>
I20260924 02:57:55.499055 7126 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:55.499097 7126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:55.499213 7126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:55.500207 7126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/data/instance:
uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba"
format_stamp: "Formatted at 2026-09-24 02:57:55 on dist-test-slave-sb6p"
server_key: "a8312f832f19c50eccad2d220c6eb749"
server_key_iv: "5698e90012eb33c15d2630d97d2dd8e2"
server_key_version: "encryptionkey@0"
I20260924 02:57:55.500566 7126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/wal/instance:
uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba"
format_stamp: "Formatted at 2026-09-24 02:57:55 on dist-test-slave-sb6p"
server_key: "a8312f832f19c50eccad2d220c6eb749"
server_key_iv: "5698e90012eb33c15d2630d97d2dd8e2"
server_key_version: "encryptionkey@0"
I20260924 02:57:55.501782 7126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 02:57:55.502414 7141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.502597 7126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:55.502676 7126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/wal
uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba"
format_stamp: "Formatted at 2026-09-24 02:57:55 on dist-test-slave-sb6p"
server_key: "a8312f832f19c50eccad2d220c6eb749"
server_key_iv: "5698e90012eb33c15d2630d97d2dd8e2"
server_key_version: "encryptionkey@0"
I20260924 02:57:55.502740 7126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:55.522579 7126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:55.524406 7126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:55.524547 7126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:55.528520 7126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36985
I20260924 02:57:55.528549 7193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36985 every 8 connection(s)
I20260924 02:57:55.528869 7126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/data/info.pb
I20260924 02:57:55.529742 7194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:55.530807 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 7126
I20260924 02:57:55.531329 7194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218674, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 02:57:55.539129 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.532060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34594 (local address 127.0.98.62:37993)
0924 02:57:55.532110 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.532112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.532121 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:57:55.532203 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.532204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.532214 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.532261 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.532773 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.533277 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.534251 (+ 974us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.534504 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.534591 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.536690 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.536692 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.536692 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.536700 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.537908 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.538285 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.538286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.538286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.538339 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.538557 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.538558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.538558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.538666 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.538720 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.538966 (+ 246us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.539040 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 02:57:55.539270 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.531895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.0.1:34594)
0924 02:57:55.532028 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.532064 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.532139 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.532284 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.532287 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.532539 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.532688 (+ 149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.532691 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.533379 (+ 688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.533380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.534181 (+ 801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.534183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.534595 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.534596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.534708 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.535193 (+ 485us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.535201 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.536611 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.537980 (+ 1369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.537986 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.537988 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.538224 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.538389 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.538390 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.538390 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.538483 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.538718 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.538723 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.538848 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.538934 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.539068 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260924 02:57:55.539847 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.531372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40518 (local address 127.0.98.60:36985)
0924 02:57:55.531530 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.531540 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.531572 (+ 32us) server_negotiation.cc:407] Connection header received
0924 02:57:55.532074 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.532083 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.532142 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.532238 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.532797 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.533713 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.534425 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.534809 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.534894 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.537203 (+ 2309us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.537209 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.537209 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.537214 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.538672 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.538979 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.538981 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.538982 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.539029 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.539185 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.539185 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.539186 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.539349 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.539500 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.539693 (+ 193us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.539751 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":65,"threads_started":1}
I20260924 02:57:55.539987 7194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 02:57:55.545379 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.540184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41113 (local address 127.0.0.1:38828)
0924 02:57:55.540225 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.540235 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.540273 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.540403 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.540403 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.540531 (+ 128us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.540569 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.540570 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.541179 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.541180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.541672 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.541673 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.542006 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.542007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.542068 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.542372 (+ 304us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.542377 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.543403 (+ 1026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.544547 (+ 1144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.544548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.544548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.544693 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.544846 (+ 153us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.544846 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.544847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.544898 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.545107 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.545108 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.545146 (+ 38us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.545205 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.545284 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 02:57:55.545385 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.540251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38828 (local address 127.0.98.61:41113)
0924 02:57:55.540291 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.540293 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.540310 (+ 17us) server_negotiation.cc:407] Connection header received
0924 02:57:55.540339 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.540340 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.540353 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.540395 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.540635 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.541115 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.541737 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.541953 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.542004 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.543476 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.543478 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.543478 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.543484 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.544491 (+ 1007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.544748 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.544748 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.544749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.544796 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.544946 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.544947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.544947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.545056 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.545109 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.545231 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.545304 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 02:57:55.545780 7194 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 02:57:55.550823 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.545977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36985 (local address 127.0.0.1:40520)
0924 02:57:55.546006 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.546017 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.546053 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.546136 (+ 83us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.546137 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.546241 (+ 104us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.546264 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.546265 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.546887 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.546888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.547356 (+ 468us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.547357 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.547636 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.547636 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.547668 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.547970 (+ 302us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.547974 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.548938 (+ 964us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.550147 (+ 1209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.550148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.550148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.550272 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.550408 (+ 136us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.550408 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.550408 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.550446 (+ 38us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.550630 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.550630 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.550660 (+ 30us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.550706 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.550769 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15}
I20260924 02:57:55.550899 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.546030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40520 (local address 127.0.98.60:36985)
0924 02:57:55.546068 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.546069 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.546075 (+ 6us) server_negotiation.cc:407] Connection header received
0924 02:57:55.546090 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.546091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.546097 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.546134 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.546307 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.546836 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.547411 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.547593 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.547632 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.549001 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.549002 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.549003 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.549008 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.550092 (+ 1084us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.550317 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.550317 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.550317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.550367 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.550479 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.550479 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.550479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.550586 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.550630 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.550721 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.550810 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":37,"server-negotiator.queue_time_us":24}
I20260924 02:57:55.551976 7194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: Bootstrap starting.
I20260924 02:57:55.552717 7194 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:55.552947 7194 log.cc:824] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:55.553568 7194 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: No bootstrap required, opened a new log
I20260924 02:57:55.555306 7194 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } }
I20260924 02:57:55.555440 7194 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:55.555474 7194 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cfa1c2eeee74f3e826cfe62bc1787ba, State: Initialized, Role: FOLLOWER
I20260924 02:57:55.555588 7194 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } }
I20260924 02:57:55.555802 7199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } } }
I20260924 02:57:55.555857 7194 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:55.555872 7199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: This master's current role is: FOLLOWER
W20260924 02:57:55.557240 7210 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 02:57:55.557292 7210 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 02:57:55.559186 7126 master_runner.cc:428] Detected that this master 0cfa1c2eeee74f3e826cfe62bc1787ba is joining an existing cluster
I20260924 02:57:55.559221 7126 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.98.60:36985
I20260924 02:57:55.566409 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.560331 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40534 (local address 127.0.98.60:36985)
0924 02:57:55.560373 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.560374 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.560658 (+ 284us) server_negotiation.cc:407] Connection header received
0924 02:57:55.560734 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.560735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.560742 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.560766 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.561190 (+ 424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.561865 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.562898 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.563164 (+ 266us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.563199 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.564304 (+ 1105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.564305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.564305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.564310 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.565424 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.565785 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.565786 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.565786 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.565824 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.565984 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.565985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.565985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.566064 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.566105 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.566295 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.566355 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 02:57:55.566658 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.560405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36985 (local address 127.0.0.1:40534)
0924 02:57:55.560602 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.560611 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.560677 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.560810 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.560810 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.561059 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.561115 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.561117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.561945 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.561946 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.562804 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.562806 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.563270 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.563271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.563337 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.563806 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.563810 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.564222 (+ 412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.565496 (+ 1274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.565497 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.565498 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.565721 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.565875 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.565876 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.565876 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.565936 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.566126 (+ 190us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.566127 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.566188 (+ 61us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.566283 (+ 95us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.566429 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":183,"mutex_wait_us":34,"thread_start_us":73,"threads_started":1}
I20260924 02:57:55.576167 7199 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:57:55.576264 7199 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } }
I20260924 02:57:55.576496 7199 leader_election.cc:288] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eac45a5647564b07822c25ba28440268 (127.0.98.62:37993), a8332978b13547c88de3e29f5b370cf3 (127.0.98.61:41113)
I20260924 02:57:55.576689 6483 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eac45a5647564b07822c25ba28440268" is_pre_election: true
I20260924 02:57:55.576694 7082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a8332978b13547c88de3e29f5b370cf3" is_pre_election: true
I20260924 02:57:55.576799 7082 raft_consensus.cc:1799] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 1 LEARNER]: Handling vote request from an unknown peer 0cfa1c2eeee74f3e826cfe62bc1787ba
I20260924 02:57:55.576849 7082 raft_consensus.cc:2396] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 1 LEARNER]: Leader pre-election vote request: Denying vote to candidate 0cfa1c2eeee74f3e826cfe62bc1787ba for term 1 because replica has last-logged OpId of term: 1 index: 5, which is greater than that of the candidate, which has last-logged OpId of term: 0 index: 0.
I20260924 02:57:55.577059 7144 leader_election.cc:302] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 0cfa1c2eeee74f3e826cfe62bc1787ba; no voters: a8332978b13547c88de3e29f5b370cf3, eac45a5647564b07822c25ba28440268
I20260924 02:57:55.577170 7199 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:55.577908 7199 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 02:57:55.586246 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.560355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34606 (local address 127.0.98.62:37993)
0924 02:57:55.560394 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.560395 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.560744 (+ 349us) server_negotiation.cc:407] Connection header received
0924 02:57:55.560825 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.560826 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.560845 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.560873 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.561225 (+ 352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.561930 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.562984 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.563275 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.563312 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.583754 (+ 20442us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.583755 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.583756 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.583765 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.585163 (+ 1398us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.585458 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.585459 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.585460 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.585516 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.585702 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.585704 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.585704 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.585863 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.585925 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.586063 (+ 138us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.586146 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 02:57:55.586259 7218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.560433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.0.1:34606)
0924 02:57:55.560698 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.560706 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.560764 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.560928 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.560929 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.561102 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.561152 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.561153 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.562006 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.562006 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.562892 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.562894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.563379 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.563380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.563441 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.583168 (+ 19727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.583172 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.583646 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.585237 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.585238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.585239 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.585399 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.585582 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.585582 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.585583 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.585634 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.585924 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.585925 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.585973 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.586033 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.586136 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":43,"thread_start_us":50,"threads_started":1}
I20260924 02:57:55.586737 7217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.560354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41113 (local address 127.0.0.1:38840)
0924 02:57:55.560542 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.560551 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.560606 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.560759 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.560760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.560977 (+ 217us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.561040 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.561041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.561945 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.561945 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.562892 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.562894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.563383 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.563384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.563441 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.583102 (+ 19661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.583106 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.583652 (+ 546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.585534 (+ 1882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.585534 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.585535 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.585712 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.585871 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.585872 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.585872 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.585921 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.586203 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.586204 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.586249 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.586532 (+ 283us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.586637 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":30,"thread_start_us":67,"threads_started":1}
I20260924 02:57:55.586748 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.560512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38840 (local address 127.0.98.61:41113)
0924 02:57:55.560550 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.560551 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.560609 (+ 58us) server_negotiation.cc:407] Connection header received
0924 02:57:55.560660 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.560661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.560671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.560718 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.561116 (+ 398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.561862 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.562984 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.563276 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.563311 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.584047 (+ 20736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.584049 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.584049 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.584056 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.585463 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.585771 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.585772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.585773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.585822 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.585972 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.585973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.585974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.586103 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.586160 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.586579 (+ 419us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.586660 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 02:57:55.591085 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 6535
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218674, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 02:57:55.596454 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:35253
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37993
--webserver_port=35971
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37993,127.0.98.61:41113
--tserver_master_addrs=127.0.98.62:37993,127.0.98.61:41113,127.0.98.60:36985 with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 02:57:55.597901 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.588277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36985 (local address 127.0.0.1:40536)
0924 02:57:55.588330 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.588339 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.588371 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.588484 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.588485 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.588633 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.588679 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.588681 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.589222 (+ 541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.589223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.590040 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.590043 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.590326 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.590326 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.590393 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.590765 (+ 372us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.590769 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.595317 (+ 4548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.596888 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.596889 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.596890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.597081 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.597260 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.597260 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.597261 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.597316 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.597581 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.597582 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.597626 (+ 44us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.597687 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.597791 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 02:57:55.598214 7196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.588310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40536 (local address 127.0.98.60:36985)
0924 02:57:55.588351 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.588353 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.588360 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:57:55.588432 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.588432 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.588439 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.588480 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.588736 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.589162 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.590109 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.590275 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.590313 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.595431 (+ 5118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.595432 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.595433 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.595440 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.596806 (+ 1366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.597150 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.597150 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.597151 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.597204 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.597368 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.597369 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.597369 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.597492 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.597550 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.598060 (+ 510us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.598130 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 02:57:55.598626 6473 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.0.98.60:36985: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba"
member_type: NON_VOTER
last_known_addr {
host: "127.0.98.60"
port: 36985
}
attrs {
promote: true
}
}
dest_uuid: "eac45a5647564b07822c25ba28440268"
cas_config_opid_index: 5
I20260924 02:57:55.598868 6473 consensus_queue.cc:237] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } }
I20260924 02:57:55.599186 7032 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0cfa1c2eeee74f3e826cfe62bc1787ba (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } } }
I20260924 02:57:55.600070 7032 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.0.98.60:36985
I20260924 02:57:55.600142 7032 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } } }
I20260924 02:57:55.600203 7032 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:55.600752 7168 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 1 FOLLOWER]: Refusing update from remote peer eac45a5647564b07822c25ba28440268: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 02:57:55.600987 7033 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:55.602946 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.599245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41113 (local address 127.0.0.1:38846)
0924 02:57:55.599398 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.599407 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.599449 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.599637 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.599638 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.599639 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:55.599674 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.599676 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.600605 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.600607 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.601802 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.601805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.602701 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.602702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.602766 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.602786 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.602832 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.602880 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138}
I20260924 02:57:55.602963 7168 raft_consensus.cc:858] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 1 NON_PARTICIPANT]: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } }
W20260924 02:57:55.603012 7168 raft_consensus.cc:1565] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 1 NON_PARTICIPANT]: Could not prepare op 'term: 1 index: 6' and following 0 ops. Status for this op: Illegal state: RaftConfig change currently pending. Only one is allowed at a time.
Committed config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } }.
Pending config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } }:
New config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } }
I20260924 02:57:55.603174 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.599334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38846 (local address 127.0.98.61:41113)
0924 02:57:55.599371 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.599372 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.599494 (+ 122us) server_negotiation.cc:407] Connection header received
0924 02:57:55.599514 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.599515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.599537 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.599587 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:55.599733 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.600479 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.601910 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.602487 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.602592 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.602669 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.603073 (+ 404us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.603113 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 02:57:55.603340 7082 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 1 LEARNER]: Refusing update from remote peer eac45a5647564b07822c25ba28440268: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260924 02:57:55.603503 7033 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260924 02:57:55.603509 7168 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 1 NON_PARTICIPANT]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER a8332978b13547c88de3e29f5b370cf3 (127.0.98.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } } }
I20260924 02:57:55.603998 7082 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 0cfa1c2eeee74f3e826cfe62bc1787ba (127.0.98.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } } }
I20260924 02:57:55.604496 7199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader eac45a5647564b07822c25ba28440268. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } } }
I20260924 02:57:55.604561 7199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 02:57:55.604589 7199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } } }
I20260924 02:57:55.604625 7199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 02:57:55.604650 7199 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } } }
I20260924 02:57:55.604681 7199 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: This master's current role is: NON_PARTICIPANT
I20260924 02:57:55.604727 7126 master.cc:537] Master@127.0.98.60:36985 shutting down...
I20260924 02:57:55.605192 7111 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eac45a5647564b07822c25ba28440268. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } } }
I20260924 02:57:55.605268 7111 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: This master's current role is: LEARNER
I20260924 02:57:55.605300 7111 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: NON_VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: true } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } } }
I20260924 02:57:55.605335 7111 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: This master's current role is: LEARNER
I20260924 02:57:55.605774 7033 raft_consensus.cc:1050] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268: attempting to promote NON_VOTER a8332978b13547c88de3e29f5b370cf3 to VOTER
I20260924 02:57:55.605952 7033 consensus_queue.cc:237] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } }
I20260924 02:57:55.606317 7082 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 1 LEARNER]: Refusing update from remote peer eac45a5647564b07822c25ba28440268: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260924 02:57:55.606472 7034 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260924 02:57:55.606667 6457 consensus_peers.cc:487] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 -> Peer 0cfa1c2eeee74f3e826cfe62bc1787ba (127.0.98.60:36985): Couldn't send request to peer 0cfa1c2eeee74f3e826cfe62bc1787ba. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260924 02:57:55.607038 7032 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a8332978b13547c88de3e29f5b370cf3 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } } }
I20260924 02:57:55.607805 7032 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } } }
I20260924 02:57:55.607888 7032 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:55.607939 7032 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } } }
I20260924 02:57:55.607975 7032 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:55.608153 7082 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, a8332978b13547c88de3e29f5b370cf3 (127.0.98.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } } }
I20260924 02:57:55.609026 7225 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } } }
I20260924 02:57:55.609102 7225 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 02:57:55.609922 7126 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 1 NON_PARTICIPANT]: Raft consensus shutting down.
I20260924 02:57:55.609998 7126 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 1 NON_PARTICIPANT]: Raft consensus is shut down!
I20260924 02:57:55.610028 7126 tablet_replica.cc:330] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: stopping tablet replica
I20260924 02:57:55.627311 7126 master.cc:552] Master@127.0.98.60:36985 shutdown complete.
I20260924 02:57:55.627381 7126 master_runner.cc:305] Clearing existing system tablet
I20260924 02:57:55.627501 7126 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:55.627547 7126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:55.628245 7126 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:55.628773 7230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.628913 7126 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:55.628975 7126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.628998 7126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/wal
uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba"
format_stamp: "Formatted at 2026-09-24 02:57:55 on dist-test-slave-sb6p"
server_key: "a8312f832f19c50eccad2d220c6eb749"
server_key_iv: "5698e90012eb33c15d2630d97d2dd8e2"
server_key_version: "encryptionkey@0"
I20260924 02:57:55.629660 7126 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:55.631214 7126 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260924 02:57:55.631282 7126 log.cc:1196] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260924 02:57:55.631518 7126 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: Deleting consensus metadata
I20260924 02:57:55.631636 7126 master_runner.cc:315] Copying system tablet from 127.0.98.62:37993
I20260924 02:57:55.632210 7126 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: tablet copy: Beginning tablet copy session from remote peer at address 127.0.98.62:37993
I20260924 02:57:55.639259 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.632570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34610 (local address 127.0.98.62:37993)
0924 02:57:55.632615 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.632617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.632666 (+ 49us) server_negotiation.cc:407] Connection header received
0924 02:57:55.632728 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.632729 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.632740 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.632813 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.633174 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.633939 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.635015 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.635315 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.635405 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.636517 (+ 1112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.636518 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.636519 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.636526 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.638040 (+ 1514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.638415 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.638416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.638417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.638480 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.638707 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.638708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.638708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.638846 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.638922 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.639086 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.639171 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 02:57:55.639274 7236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.632502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.0.1:34610)
0924 02:57:55.632611 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.632621 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.632675 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.632827 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.632828 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.633035 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.633081 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.633083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.634028 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.634029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.634922 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.634925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.635400 (+ 475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.635401 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.635461 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.635959 (+ 498us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.635964 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.636399 (+ 435us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.638128 (+ 1729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.638129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.638130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.638335 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.638575 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.638576 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.638576 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.638638 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.638918 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.638920 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.638972 (+ 52us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.639049 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.639158 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":57,"threads_started":1}
I20260924 02:57:55.639976 6493 tablet_copy_service.cc:140] P eac45a5647564b07822c25ba28440268: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 0cfa1c2eeee74f3e826cfe62bc1787ba ({username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34610)
I20260924 02:57:55.640039 6493 tablet_copy_service.cc:161] P eac45a5647564b07822c25ba28440268: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 0cfa1c2eeee74f3e826cfe62bc1787ba at {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34610: session id = 0cfa1c2eeee74f3e826cfe62bc1787ba-00000000000000000000000000000000
I20260924 02:57:55.640657 6493 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268: Tablet Copy: opened 0 blocks and 1 log segments
I20260924 02:57:55.641127 7126 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260924 02:57:55.642439 7126 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: tablet copy: Starting download of 0 data blocks...
I20260924 02:57:55.642634 7126 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: tablet copy: Starting download of 1 WAL segments...
I20260924 02:57:55.643278 7126 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260924 02:57:55.644130 6493 tablet_copy_service.cc:342] P eac45a5647564b07822c25ba28440268: Request end of tablet copy session 0cfa1c2eeee74f3e826cfe62bc1787ba-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34610
I20260924 02:57:55.644186 6493 tablet_copy_service.cc:433] P eac45a5647564b07822c25ba28440268: ending tablet copy session 0cfa1c2eeee74f3e826cfe62bc1787ba-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 0cfa1c2eeee74f3e826cfe62bc1787ba
I20260924 02:57:55.647019 7126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:55.648319 7240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:55.648362 7237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:55.648506 7126 server_base.cc:1034] running on GCE node
W20260924 02:57:55.648587 7238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:55.648721 7126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:55.648860 7126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:55.649986 7126 hybrid_clock.cc:648] HybridClock initialized: now 1790218675649977 us; error 27 us; skew 500 ppm
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:55.651407 7126 init.cc:377] Logged in from keytab as oryx/127.0.98.60@KRBTEST.COM (short username oryx)
I20260924 02:57:55.651640 7126 webserver.cc:460] Webserver started at http://127.0.98.60:46879/ using document root <none> and password file <none>
I20260924 02:57:55.651744 7126 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:55.651775 7126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:55.652438 7126 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:55.652959 7247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.653053 7126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.653090 7126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/wal
uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba"
format_stamp: "Formatted at 2026-09-24 02:57:55 on dist-test-slave-sb6p"
server_key: "a8312f832f19c50eccad2d220c6eb749"
server_key_iv: "5698e90012eb33c15d2630d97d2dd8e2"
server_key_version: "encryptionkey@0"
I20260924 02:57:55.653141 7126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:55.665792 7126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:55.667557 7126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:55.670886 7126 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:36985
I20260924 02:57:55.671195 7126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/master-2/data/info.pb
I20260924 02:57:55.672127 7300 sys_catalog.cc:263] Verifying existing consensus state
I20260924 02:57:55.672627 7300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: Bootstrap starting.
I20260924 02:57:55.673717 7299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:36985 every 8 connection(s)
I20260924 02:57:55.674998 7300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 02:57:55.675290 7300 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: Bootstrap complete.
I20260924 02:57:55.675550 7300 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } }
I20260924 02:57:55.675644 7300 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 0cfa1c2eeee74f3e826cfe62bc1787ba, State: Initialized, Role: LEARNER
I20260924 02:57:55.675693 7300 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } }
I20260924 02:57:55.675863 7300 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:55.675869 7303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } } }
I20260924 02:57:55.676097 7303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: This master's current role is: LEARNER
I20260924 02:57:55.677321 7314 catalog_manager.cc:1259] Loaded cluster ID: 58c36acd554e4645bd70b6b2991339b9
I20260924 02:57:55.677425 7314 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: loading cluster ID for follower catalog manager: success
I20260924 02:57:55.678596 7314 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: acquiring CA information for follower catalog manager: success
I20260924 02:57:55.678891 7314 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260924 02:57:55.694861 7219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:55.695063 7219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:55.695096 7219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:55.696234 7219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:55.696276 7219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:55.696310 7219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:57:55.697762 7219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:35253
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35971
--tserver_master_addrs=127.0.98.62:37993,127.0.98.61:41113,127.0.98.60:36985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:55.697983 7219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:55.698180 7219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:55.702759 7322 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:55.702877 7320 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:55.702777 7219 server_base.cc:1034] running on GCE node
W20260924 02:57:55.702797 7319 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:55.703172 7219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:55.703397 7219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:55.704522 7219 hybrid_clock.cc:648] HybridClock initialized: now 1790218675704514 us; error 26 us; skew 500 ppm
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:55.707297 7219 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 02:57:55.707680 7219 webserver.cc:460] Webserver started at http://127.0.98.1:35971/ using document root <none> and password file <none>
I20260924 02:57:55.707888 7219 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:55.707949 7219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:55.709199 7219 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:55.709682 7329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.709827 7219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.709908 7219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "6cda0e259079447197b80ec4eb9acf9f"
format_stamp: "Formatted at 2026-09-24 02:57:54 on dist-test-slave-sb6p"
server_key: "e8eedcd57a90e08a205b1dbe26e6547e"
server_key_iv: "ca1f65a1f7760d6a1caf48d9db1424d5"
server_key_version: "encryptionkey@0"
I20260924 02:57:55.710222 7219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:55.726565 7219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:55.728629 7219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:55.728763 7219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:55.728976 7219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:55.729251 7219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:55.729283 7219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.729311 7219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:55.729332 7219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.734889 7219 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:35253
I20260924 02:57:55.734933 7442 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:35253 every 8 connection(s)
I20260924 02:57:55.735255 7219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:57:55.741328 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 7219
I20260924 02:57:55.741480 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 6669
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
I20260924 02:57:55.746413 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.736062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51881 (local address 127.0.98.61:41113)
0924 02:57:55.736106 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.736108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.736551 (+ 443us) server_negotiation.cc:407] Connection header received
0924 02:57:55.736939 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.736940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.736952 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.736999 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.737792 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.738559 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.739576 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.739832 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.739900 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.743492 (+ 3592us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.743494 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.743494 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.743502 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.744895 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.745406 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.745407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.745408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.745463 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.745743 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.745743 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.745744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.745899 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.745942 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.746252 (+ 310us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.746328 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 02:57:55.746417 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.735993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60789 (local address 127.0.98.62:37993)
0924 02:57:55.736162 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.736164 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.736406 (+ 242us) server_negotiation.cc:407] Connection header received
0924 02:57:55.737020 (+ 614us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.737022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.737032 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.737076 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.737759 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.738550 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.739749 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.740006 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.740037 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.743630 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.743631 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.743632 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.743639 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.745057 (+ 1418us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.745482 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.745483 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.745483 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.745535 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.745729 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.745730 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.745730 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.745847 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.745885 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.746242 (+ 357us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.746317 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260924 02:57:55.746806 7448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.735970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.1:60789)
0924 02:57:55.736255 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.736300 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.736844 (+ 544us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.737119 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.737123 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.737479 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.737675 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.737679 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.738620 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.738621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.739652 (+ 1031us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.739655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.740096 (+ 441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.740097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.740139 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.740650 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.740654 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.743545 (+ 2891us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.745156 (+ 1611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.745157 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.745157 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.745407 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.745595 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.745596 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.745596 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.745673 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.745923 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.745928 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.746087 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.746545 (+ 458us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.746641 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":90,"thread_start_us":65,"threads_started":1}
I20260924 02:57:55.746994 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.735970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41113 (local address 127.0.98.1:51881)
0924 02:57:55.736494 (+ 524us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.736502 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.736846 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.737385 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.737385 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.737541 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.737677 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.737682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.738646 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.738647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.739497 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.739499 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.739961 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.739962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.740092 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.740605 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.740612 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.743409 (+ 2797us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.744969 (+ 1560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.744969 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.744970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.745322 (+ 352us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.745529 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.745529 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.745530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.745670 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.745996 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.745997 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.746088 (+ 91us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.746663 (+ 575us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.746759 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"mutex_wait_us":221,"thread_start_us":66,"threads_started":1}
I20260924 02:57:55.747023 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.736064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49755 (local address 127.0.98.60:36985)
0924 02:57:55.736206 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.736208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.736398 (+ 190us) server_negotiation.cc:407] Connection header received
0924 02:57:55.737133 (+ 735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.737134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.737147 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.737205 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.737770 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.738420 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.739877 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.740156 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.740223 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.743329 (+ 3106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.743330 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.743330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.743338 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.744793 (+ 1455us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.745407 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.745408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.745408 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.745464 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.745735 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.745736 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.745737 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.745902 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.745954 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.746752 (+ 798us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.746831 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":55,"threads_started":1}
I20260924 02:57:55.747249 7450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.735970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36985 (local address 127.0.98.1:49755)
0924 02:57:55.736255 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.736300 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.736837 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.737225 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.737226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.737502 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.737677 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.737682 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.738533 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.738534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.739809 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.739811 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.740234 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.740234 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.740645 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.741043 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.741047 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.743244 (+ 2197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.744885 (+ 1641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.744892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.744894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.745322 (+ 428us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.745527 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.745527 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.745528 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.745670 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.745978 (+ 308us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.745978 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.746089 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.746206 (+ 117us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.746368 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":111,"thread_start_us":76,"threads_started":1}
I20260924 02:57:55.747650 7443 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36985
I20260924 02:57:55.747742 7443 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:55.747915 7443 heartbeater.cc:511] Master 127.0.98.60:36985 requested a full tablet report, sending...
I20260924 02:57:55.748402 7444 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37993
I20260924 02:57:55.748415 7264 ts_manager.cc:194] Registered new tserver with Master: 6cda0e259079447197b80ec4eb9acf9f (127.0.98.1:35253)
I20260924 02:57:55.748413 7445 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41113
I20260924 02:57:55.748461 7444 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:55.748479 7445 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:55.748520 7444 heartbeater.cc:511] Master 127.0.98.62:37993 requested a full tablet report, sending...
I20260924 02:57:55.748539 7445 heartbeater.cc:511] Master 127.0.98.61:41113 requested a full tablet report, sending...
I20260924 02:57:55.748801 6473 ts_manager.cc:194] Re-registered known tserver with Master: 6cda0e259079447197b80ec4eb9acf9f (127.0.98.1:35253)
I20260924 02:57:55.748893 7072 ts_manager.cc:194] Registered new tserver with Master: 6cda0e259079447197b80ec4eb9acf9f (127.0.98.1:35253)
I20260924 02:57:55.749359 6473 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60789
I20260924 02:57:55.751596 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:45269
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37993
--webserver_port=33271
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37993,127.0.98.61:41113
--tserver_master_addrs=127.0.98.62:37993,127.0.98.61:41113,127.0.98.60:36985 with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:55.831568 7451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:55.831763 7451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:55.831791 7451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:55.832911 7451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:55.832950 7451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:55.832985 7451 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:57:55.834390 7451 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:45269
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=33271
--tserver_master_addrs=127.0.98.62:37993,127.0.98.61:41113,127.0.98.60:36985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7451
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:55.834649 7451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:55.834857 7451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:55.839560 7456 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:55.839556 7457 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:55.839679 7451 server_base.cc:1034] running on GCE node
W20260924 02:57:55.839639 7459 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:55.839977 7451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:55.840165 7451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:55.841289 7451 hybrid_clock.cc:648] HybridClock initialized: now 1790218675841279 us; error 27 us; skew 500 ppm
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:55.843523 7451 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 02:57:55.843915 7451 webserver.cc:460] Webserver started at http://127.0.98.2:33271/ using document root <none> and password file <none>
I20260924 02:57:55.844092 7451 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:55.844141 7451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:55.845355 7451 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 02:57:55.845863 7466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.845995 7451 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:55.846071 7451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "50ada8ae64f647f8826a849c1377ac53"
format_stamp: "Formatted at 2026-09-24 02:57:54 on dist-test-slave-sb6p"
server_key: "093946ac96722bf1bad8bdcd53d8c459"
server_key_iv: "1761d79f8fb3243d31df33d00c7e0fc3"
server_key_version: "encryptionkey@0"
I20260924 02:57:55.846398 7451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:55.861863 7451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:55.863552 7451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:55.863663 7451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:55.863883 7451 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:55.864151 7451 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:55.864180 7451 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.864202 7451 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:55.864224 7451 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.869601 7451 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45269
I20260924 02:57:55.869652 7579 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45269 every 8 connection(s)
I20260924 02:57:55.869951 7451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 02:57:55.875921 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 7451
I20260924 02:57:55.876070 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 6803
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 02:57:55.881460 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.870652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44585 (local address 127.0.98.61:41113)
0924 02:57:55.870798 (+ 146us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.870800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.871172 (+ 372us) server_negotiation.cc:407] Connection header received
0924 02:57:55.871968 (+ 796us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.871970 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.871981 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.872008 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.872798 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.873572 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.875570 (+ 1998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.875799 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.875875 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.878779 (+ 2904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.878780 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.878781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.878788 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.880197 (+ 1409us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.880540 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.880541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.880542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.880603 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.880766 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.880767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.880767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.880901 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.880949 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.881290 (+ 341us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.881369 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260924 02:57:55.881472 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.870703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41685 (local address 127.0.98.62:37993)
0924 02:57:55.870879 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.870881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.871054 (+ 173us) server_negotiation.cc:407] Connection header received
0924 02:57:55.871959 (+ 905us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.871960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.871970 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.871998 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.872770 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.873535 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.875210 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.875484 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.875532 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.878442 (+ 2910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.878443 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.878444 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.878451 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.879969 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.880438 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.880440 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.880440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.880502 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.880783 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.880784 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.880784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.880914 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.880958 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.881293 (+ 335us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.881383 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260924 02:57:55.882006 7586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.870593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41113 (local address 127.0.98.2:44585)
0924 02:57:55.871100 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.871107 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.871875 (+ 768us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.872108 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.872109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.872470 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.872689 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.872695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.873641 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.873641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.874969 (+ 1328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.874972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.875881 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.875882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.875954 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.876618 (+ 664us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.876622 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.878699 (+ 2077us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.880279 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.880280 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.880280 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.880468 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.880658 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.880658 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.880659 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.880711 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.880980 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.880986 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.881145 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.881607 (+ 462us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.881776 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":485,"mutex_wait_us":55,"spinlock_wait_cycles":7936,"thread_start_us":98,"threads_started":1}
I20260924 02:57:55.882202 7585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.870593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36985 (local address 127.0.98.2:38975)
0924 02:57:55.870935 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.870953 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.871860 (+ 907us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.872108 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.872109 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.872479 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.872689 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.872695 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.874243 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.874244 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.874968 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.874971 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.875398 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.875399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.875501 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.876322 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.876326 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.878935 (+ 2609us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.881216 (+ 2281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.881217 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.881217 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.881413 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.881609 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.881609 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.881610 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.881667 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.881921 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.881922 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.881958 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.882032 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.882111 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319,"mutex_wait_us":8,"thread_start_us":93,"threads_started":1}
I20260924 02:57:55.882233 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.870754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:38975 (local address 127.0.98.60:36985)
0924 02:57:55.870853 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:55.870855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:55.871033 (+ 178us) server_negotiation.cc:407] Connection header received
0924 02:57:55.871950 (+ 917us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:55.871952 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:55.871969 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:55.872022 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:55.873654 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.874172 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.875038 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.875322 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.875390 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.879430 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:55.879431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:55.879431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:55.879635 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:55.881138 (+ 1503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.881489 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.881490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.881490 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.881543 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:55.881727 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:55.881727 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:55.881728 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:55.881839 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:55.881893 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:55.882075 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 02:57:55.882153 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260924 02:57:55.882486 7582 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41113
I20260924 02:57:55.882596 7582 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:55.882777 7582 heartbeater.cc:511] Master 127.0.98.61:41113 requested a full tablet report, sending...
I20260924 02:57:55.882786 7580 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36985
I20260924 02:57:55.882850 7580 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:55.882926 7580 heartbeater.cc:511] Master 127.0.98.60:36985 requested a full tablet report, sending...
I20260924 02:57:55.883201 7072 ts_manager.cc:194] Registered new tserver with Master: 50ada8ae64f647f8826a849c1377ac53 (127.0.98.2:45269)
I20260924 02:57:55.883224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:43571
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37993
--webserver_port=45695
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.98.62:37993,127.0.98.61:41113
--tserver_master_addrs=127.0.98.62:37993,127.0.98.61:41113,127.0.98.60:36985 with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 02:57:55.883447 7584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:55.870592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.2:41685)
0924 02:57:55.870922 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:55.870952 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:55.871855 (+ 903us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:55.872103 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:55.872106 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:55.872465 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:55.872688 (+ 223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.872692 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.873621 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.873622 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.874969 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:55.874972 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:55.875544 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:55.875545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:55.875598 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:55.876116 (+ 518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:55.876125 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:55.878336 (+ 2211us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:55.880052 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.880059 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.880061 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.880333 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.880577 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:55.880577 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:55.880578 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:55.880705 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:55.881005 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:55.881005 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:55.881145 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 02:57:55.881876 (+ 731us) client_negotiation.cc:240] Negotiation successful
0924 02:57:55.881977 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"mutex_wait_us":29,"thread_start_us":98,"threads_started":1}
I20260924 02:57:55.884451 7264 ts_manager.cc:194] Registered new tserver with Master: 50ada8ae64f647f8826a849c1377ac53 (127.0.98.2:45269)
I20260924 02:57:55.884626 7581 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37993
I20260924 02:57:55.884675 7581 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:55.884729 7581 heartbeater.cc:511] Master 127.0.98.62:37993 requested a full tablet report, sending...
I20260924 02:57:55.884945 6473 ts_manager.cc:194] Re-registered known tserver with Master: 50ada8ae64f647f8826a849c1377ac53 (127.0.98.2:45269)
I20260924 02:57:55.885403 6473 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41685
W20260924 02:57:55.961529 7587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:55.961719 7587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:55.961740 7587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:55.962809 7587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:55.962842 7587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:55.962867 7587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:57:55.964226 7587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:43571
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=45695
--tserver_master_addrs=127.0.98.62:37993,127.0.98.61:41113,127.0.98.60:36985
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:55.964426 7587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:55.964604 7587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:55.969439 7593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:55.969491 7587 server_base.cc:1034] running on GCE node
W20260924 02:57:55.969446 7592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:55.969449 7595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:55.969928 7587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:55.970126 7587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:55.971253 7587 hybrid_clock.cc:648] HybridClock initialized: now 1790218675971241 us; error 26 us; skew 500 ppm
Sep 24 02:57:55 dist-test-slave-sb6p krb5kdc[6428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:55.973512 7587 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 02:57:55.973991 7587 webserver.cc:460] Webserver started at http://127.0.98.3:45695/ using document root <none> and password file <none>
I20260924 02:57:55.974205 7587 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:55.974262 7587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:55.975703 7587 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:55.976179 7602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.976363 7587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:55.976446 7587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "1631c5c9efc94c31975d48812ba9f320"
format_stamp: "Formatted at 2026-09-24 02:57:55 on dist-test-slave-sb6p"
server_key: "4526473c018e15d9b9a1635734ce862a"
server_key_iv: "51c933242f14a759228110c755436dd4"
server_key_version: "encryptionkey@0"
I20260924 02:57:55.976796 7587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:55.991735 7587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:55.993353 7587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:55.993475 7587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:55.993665 7587 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:57:55.993928 7587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:55.993958 7587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.993985 7587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:55.994009 7587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:55.999215 7587 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43571
I20260924 02:57:55.999266 7715 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43571 every 8 connection(s)
I20260924 02:57:55.999573 7587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 02:57:56.007752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 7587
Sep 24 02:57:56 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
Sep 24 02:57:56 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.60@KRBTEST.COM
Sep 24 02:57:56 dist-test-slave-sb6p krb5kdc[6428](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218675, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.61@KRBTEST.COM
I20260924 02:57:56.015431 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.000262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:51667 (local address 127.0.98.60:36985)
0924 02:57:56.000359 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:56.000361 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:56.000819 (+ 458us) server_negotiation.cc:407] Connection header received
0924 02:57:56.001441 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:56.001442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:56.001454 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:56.001484 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:56.002192 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.002991 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.004274 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.004516 (+ 242us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.004563 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.012149 (+ 7586us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:56.012151 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:56.012151 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:56.012159 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:56.013871 (+ 1712us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.014326 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.014327 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.014328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.014398 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.014695 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.014696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.014696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.014821 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:56.014876 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:56.015258 (+ 382us) server_negotiation.cc:299] Negotiation successful
0924 02:57:56.015328 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260924 02:57:56.015738 7722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.000196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36985 (local address 127.0.98.3:51667)
0924 02:57:56.000761 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:56.000768 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:56.001333 (+ 565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:56.001558 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:56.001563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:56.001938 (+ 375us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:56.002085 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.002091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.003077 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.003078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.004191 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.004192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.004586 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.004587 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.004637 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.009681 (+ 5044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:56.009691 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:56.012059 (+ 2368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:56.013955 (+ 1896us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:56.013961 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:56.013964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:56.014246 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:56.014462 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:56.014463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:56.014463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:56.014620 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:56.014901 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:56.014908 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:56.015071 (+ 163us) client_negotiation.cc:769] Sending connection context
0924 02:57:56.015171 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 02:57:56.015621 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"mutex_wait_us":609,"thread_start_us":56,"threads_started":1}
I20260924 02:57:56.016026 6523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.000443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57085 (local address 127.0.98.62:37993)
0924 02:57:56.000497 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:56.000499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:56.000722 (+ 223us) server_negotiation.cc:407] Connection header received
0924 02:57:56.001409 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:56.001411 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:56.001423 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:56.001475 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:56.002165 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.002957 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.004605 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.004839 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.004908 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.012557 (+ 7649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:56.012558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:56.012559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:56.012566 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:56.014153 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.014547 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.014548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.014549 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.014602 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.014786 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.014787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.014787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.014899 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:56.014955 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:56.015780 (+ 825us) server_negotiation.cc:299] Negotiation successful
0924 02:57:56.015844 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 02:57:56.015626 7721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.000197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.3:57085)
0924 02:57:56.000654 (+ 457us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:56.000662 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:56.001325 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:56.001558 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:56.001562 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:56.001895 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:56.002084 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.002088 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.003077 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.003078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.004184 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.004186 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.004913 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.004914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.004964 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.009888 (+ 4924us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:56.009892 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:56.012480 (+ 2588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:56.014242 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:56.014243 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:56.014244 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:56.014452 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:56.014669 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:56.014670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:56.014670 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:56.014725 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:56.014979 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:56.014980 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:56.015070 (+ 90us) client_negotiation.cc:769] Sending connection context
0924 02:57:56.015170 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 02:57:56.015342 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"mutex_wait_us":41,"thread_start_us":91,"threads_started":1}
I20260924 02:57:56.016242 7716 heartbeater.cc:347] Connected to a master server at 127.0.98.60:36985
I20260924 02:57:56.016320 7716 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:56.016498 7716 heartbeater.cc:511] Master 127.0.98.60:36985 requested a full tablet report, sending...
I20260924 02:57:56.016705 7717 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37993
I20260924 02:57:56.016753 7717 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:56.016836 7717 heartbeater.cc:511] Master 127.0.98.62:37993 requested a full tablet report, sending...
I20260924 02:57:56.016948 7264 ts_manager.cc:194] Registered new tserver with Master: 1631c5c9efc94c31975d48812ba9f320 (127.0.98.3:43571)
I20260924 02:57:56.017060 6473 ts_manager.cc:194] Re-registered known tserver with Master: 1631c5c9efc94c31975d48812ba9f320 (127.0.98.3:43571)
I20260924 02:57:56.017525 6473 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:57085
I20260924 02:57:56.017555 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.000196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41113 (local address 127.0.98.3:59559)
0924 02:57:56.000372 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:56.000411 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:56.001317 (+ 906us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:56.001558 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:56.001563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:56.001923 (+ 360us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:56.002085 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.002091 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.003124 (+ 1033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.003125 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.004016 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.004018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.004379 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.004379 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.004492 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.009681 (+ 5189us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:56.009691 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:56.012675 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:56.015399 (+ 2724us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:56.015400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:56.015401 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:56.015576 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:56.015771 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:56.015771 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:56.015771 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:56.015824 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:56.017266 (+ 1442us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:56.017267 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:56.017306 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 02:57:56.017370 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 02:57:56.017459 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":101,"thread_start_us":92,"threads_started":1}
I20260924 02:57:56.017620 7123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.000268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59559 (local address 127.0.98.61:41113)
0924 02:57:56.000503 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:56.000505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:56.000514 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:57:56.001442 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:56.001443 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:56.001454 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:56.001484 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:56.002575 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.003058 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.004094 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.004302 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.004364 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.013338 (+ 8974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:56.013339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:56.013340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:56.013347 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:56.015322 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.015652 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.015653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.015653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.015705 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.015906 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.015907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.015908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.016032 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:56.016069 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:56.017433 (+ 1364us) server_negotiation.cc:299] Negotiation successful
0924 02:57:56.017513 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20260924 02:57:56.018260 7718 heartbeater.cc:347] Connected to a master server at 127.0.98.61:41113
I20260924 02:57:56.018298 7718 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:56.018548 7718 heartbeater.cc:511] Master 127.0.98.61:41113 requested a full tablet report, sending...
I20260924 02:57:56.019011 7072 ts_manager.cc:194] Registered new tserver with Master: 1631c5c9efc94c31975d48812ba9f320 (127.0.98.3:43571)
I20260924 02:57:56.019184 7729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.008726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34624 (local address 127.0.98.62:37993)
0924 02:57:56.008963 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:56.008965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:56.009413 (+ 448us) server_negotiation.cc:407] Connection header received
0924 02:57:56.009459 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:56.009460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:56.009474 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:56.009538 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:56.010262 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.011040 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.012148 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.012424 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.012468 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.014725 (+ 2257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:56.014726 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:56.014727 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:56.014732 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:56.016373 (+ 1641us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.016793 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.016794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.016794 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.016874 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.017067 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.017068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.017068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.018768 (+ 1700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:56.018877 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:56.019006 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 02:57:56.019089 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211,"thread_start_us":63,"threads_started":1}
I20260924 02:57:56.022032 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.008833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38856 (local address 127.0.98.61:41113)
0924 02:57:56.009019 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:56.009021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:56.009028 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:57:56.009060 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:56.009061 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:56.009071 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:56.009128 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:56.010377 (+ 1249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.011038 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.011730 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.012030 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.012156 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.019261 (+ 7105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:56.019262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:56.019262 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:56.019268 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:56.020948 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.021331 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.021332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.021333 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.021392 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.021560 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.021561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.021561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.021677 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:56.021771 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:56.021861 (+ 90us) server_negotiation.cc:299] Negotiation successful
0924 02:57:56.021935 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":70,"threads_started":1}
I20260924 02:57:56.022310 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.008828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40550 (local address 127.0.98.60:36985)
0924 02:57:56.009105 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:56.009107 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:56.009113 (+ 6us) server_negotiation.cc:407] Connection header received
0924 02:57:56.009132 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:56.009132 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:56.009142 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:56.009197 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:56.012882 (+ 3685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.013627 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.017509 (+ 3882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.017790 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.017853 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.019216 (+ 1363us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:56.019218 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:56.019218 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:56.019224 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:56.020796 (+ 1572us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.021354 (+ 558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.021355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.021355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.021406 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.021649 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.021650 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.021650 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.021761 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:56.021844 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:56.022161 (+ 317us) server_negotiation.cc:299] Negotiation successful
0924 02:57:56.022231 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":260,"thread_start_us":77,"threads_started":1}
I20260924 02:57:56.021366 6473 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 02:57:56.023396 6473 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:57:56.030360 7737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.026738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35253 (local address 127.0.0.1:60238)
0924 02:57:56.026864 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:56.026872 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:56.026905 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:56.027414 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:56.027415 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:56.027418 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:56.027479 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.027481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.028403 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.028404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.029182 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.029184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.030149 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.030149 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.030206 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.030210 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:57:56.030253 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 02:57:56.030291 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":90,"threads_started":1}
I20260924 02:57:56.030390 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.027131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60238 (local address 127.0.98.1:35253)
0924 02:57:56.027276 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:56.027281 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:56.027291 (+ 10us) server_negotiation.cc:407] Connection header received
0924 02:57:56.027330 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:56.027332 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:56.027349 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:56.027420 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:56.027548 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.028338 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.029277 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.029980 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.030045 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.030120 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:56.030282 (+ 162us) server_negotiation.cc:299] Negotiation successful
0924 02:57:56.030319 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":67,"threads_started":1}
I20260924 02:57:56.030838 7738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.026746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43571 (local address 127.0.0.1:59850)
0924 02:57:56.026954 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:56.026960 (+ 6us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:56.026983 (+ 23us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:56.027528 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:56.027529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:56.027531 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:56.027571 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.027573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.028743 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.028744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.029649 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.029651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.030412 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.030413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.030467 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.030471 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:57:56.030537 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 02:57:56.030767 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":200,"thread_start_us":82,"threads_started":1}
I20260924 02:57:56.031129 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.026976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59850 (local address 127.0.98.3:43571)
0924 02:57:56.027333 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:56.027336 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:56.027345 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:57:56.027379 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:56.027381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:56.027396 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:56.027469 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:56.027631 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.028656 (+ 1025us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.029715 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.030243 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.030307 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.030373 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:56.031012 (+ 639us) server_negotiation.cc:299] Negotiation successful
0924 02:57:56.031055 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":78,"threads_started":1}
I20260924 02:57:56.031306 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.026687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45269 (local address 127.0.0.1:34094)
0924 02:57:56.026727 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:56.026739 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:56.026777 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:56.027858 (+ 1081us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:56.027859 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:56.027861 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:56.027891 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.027893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.029028 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.029029 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.030028 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.030030 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.031104 (+ 1074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.031106 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.031165 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.031168 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:56.031215 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 02:57:56.031251 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 02:57:56.031167 7377 tablet_service.cc:1467] Processing CreateTablet for tablet 68ec6b81e4fe48839ba9d8fe6d6b79ac (DEFAULT_TABLE table=test-table [id=a6ab94a032934c539949a474c21ff32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:56.031570 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.026961 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34094 (local address 127.0.98.2:45269)
0924 02:57:56.027694 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:56.027698 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:56.027707 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:57:56.027741 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:56.027743 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:56.027757 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:56.027833 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:56.027951 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.028954 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.030102 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.030996 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.031082 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.031430 (+ 348us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:56.031458 (+ 28us) server_negotiation.cc:299] Negotiation successful
0924 02:57:56.031499 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":701,"thread_start_us":87,"threads_started":1}
I20260924 02:57:56.031622 7377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ec6b81e4fe48839ba9d8fe6d6b79ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:56.031745 7650 tablet_service.cc:1467] Processing CreateTablet for tablet 68ec6b81e4fe48839ba9d8fe6d6b79ac (DEFAULT_TABLE table=test-table [id=a6ab94a032934c539949a474c21ff32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:56.032022 7650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ec6b81e4fe48839ba9d8fe6d6b79ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:56.032130 7514 tablet_service.cc:1467] Processing CreateTablet for tablet 68ec6b81e4fe48839ba9d8fe6d6b79ac (DEFAULT_TABLE table=test-table [id=a6ab94a032934c539949a474c21ff32f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:56.032413 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 68ec6b81e4fe48839ba9d8fe6d6b79ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:56.034315 7742 tablet_bootstrap.cc:492] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320: Bootstrap starting.
I20260924 02:57:56.035116 7743 tablet_bootstrap.cc:492] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f: Bootstrap starting.
I20260924 02:57:56.035491 7742 tablet_bootstrap.cc:654] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:56.035478 7744 tablet_bootstrap.cc:492] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53: Bootstrap starting.
I20260924 02:57:56.035657 7743 tablet_bootstrap.cc:654] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:56.035766 7742 log.cc:824] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:56.035948 7743 log.cc:824] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:56.036003 7744 tablet_bootstrap.cc:654] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:56.036302 7744 log.cc:824] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:56.037012 7744 tablet_bootstrap.cc:492] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53: No bootstrap required, opened a new log
I20260924 02:57:56.037075 7744 ts_tablet_manager.cc:1397] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:56.037488 7743 tablet_bootstrap.cc:492] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f: No bootstrap required, opened a new log
I20260924 02:57:56.037540 7743 ts_tablet_manager.cc:1397] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:56.038432 7744 raft_consensus.cc:348] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } }
I20260924 02:57:56.038622 7744 raft_consensus.cc:374] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:56.038663 7744 raft_consensus.cc:729] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ada8ae64f647f8826a849c1377ac53, State: Initialized, Role: FOLLOWER
I20260924 02:57:56.038770 7744 consensus_queue.cc:260] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } }
I20260924 02:57:56.038972 7742 tablet_bootstrap.cc:492] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320: No bootstrap required, opened a new log
I20260924 02:57:56.039004 7744 ts_tablet_manager.cc:1428] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 02:57:56.039037 7742 ts_tablet_manager.cc:1397] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 02:57:56.039116 7581 heartbeater.cc:503] Master 127.0.98.62:37993 was elected leader, sending a full tablet report...
I20260924 02:57:56.039448 7743 raft_consensus.cc:348] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } }
I20260924 02:57:56.039592 7743 raft_consensus.cc:374] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:56.039628 7743 raft_consensus.cc:729] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cda0e259079447197b80ec4eb9acf9f, State: Initialized, Role: FOLLOWER
I20260924 02:57:56.039729 7743 consensus_queue.cc:260] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } }
I20260924 02:57:56.039958 7743 ts_tablet_manager.cc:1428] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 02:57:56.040220 7444 heartbeater.cc:503] Master 127.0.98.62:37993 was elected leader, sending a full tablet report...
I20260924 02:57:56.040962 7742 raft_consensus.cc:348] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } }
I20260924 02:57:56.041109 7742 raft_consensus.cc:374] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:56.041165 7742 raft_consensus.cc:729] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1631c5c9efc94c31975d48812ba9f320, State: Initialized, Role: FOLLOWER
I20260924 02:57:56.041270 7742 consensus_queue.cc:260] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } }
I20260924 02:57:56.041514 7742 ts_tablet_manager.cc:1428] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 02:57:56.041642 7717 heartbeater.cc:503] Master 127.0.98.62:37993 was elected leader, sending a full tablet report...
I20260924 02:57:56.071621 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.069202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40556 (local address 127.0.98.60:36985)
0924 02:57:56.069266 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:56.069268 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:56.069277 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:57:56.069306 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:56.069307 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:56.069328 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:56.069379 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:56.069513 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.070005 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.070905 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.071298 (+ 393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.071362 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.071446 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:56.071513 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 02:57:56.071557 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 02:57:56.071622 7031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.069139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36985 (local address 127.0.0.1:40556)
0924 02:57:56.069194 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:56.069204 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:56.069244 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:56.069399 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:56.069400 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:56.069402 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:56.069455 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.069456 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.070074 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.070076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.070827 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.070830 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.071381 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.071382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.071441 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.071445 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:57:56.071491 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 02:57:56.071538 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 02:57:56.071966 7274 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.9] Dedup: 1.7->[1.8-1.9]
I20260924 02:57:56.072317 7303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader eac45a5647564b07822c25ba28440268. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } } }
I20260924 02:57:56.072451 7303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: This master's current role is: LEARNER
W20260924 02:57:56.120620 7583 tablet.cc:2369] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 02:57:56.146883 7750 raft_consensus.cc:482] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:57:56.147045 7750 raft_consensus.cc:504] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } }
I20260924 02:57:56.147429 7750 leader_election.cc:288] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 50ada8ae64f647f8826a849c1377ac53 (127.0.98.2:45269), 6cda0e259079447197b80ec4eb9acf9f (127.0.98.1:35253)
I20260924 02:57:56.150231 7721 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.147598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35253 (local address 127.0.98.3:46295)
0924 02:57:56.147774 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:56.147784 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:56.147834 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:56.147973 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:56.147974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:56.147975 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:56.148030 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.148032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.148605 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.148606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.149400 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.149402 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.149983 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.149984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.150070 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.150073 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:56.150116 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 02:57:56.150158 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"mutex_wait_us":154}
I20260924 02:57:56.150231 7741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.147643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:46295 (local address 127.0.98.1:35253)
0924 02:57:56.147698 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:56.147700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:56.147835 (+ 135us) server_negotiation.cc:407] Connection header received
0924 02:57:56.147903 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:56.147904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:56.147915 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:56.147968 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:56.148087 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.148533 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.149493 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.149914 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.149976 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.149988 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:56.150129 (+ 141us) server_negotiation.cc:299] Negotiation successful
0924 02:57:56.150167 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 02:57:56.150549 7739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.147631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43651 (local address 127.0.98.2:45269)
0924 02:57:56.147699 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:56.147701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:56.147711 (+ 10us) server_negotiation.cc:407] Connection header received
0924 02:57:56.147792 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:56.147793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:56.147806 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:56.147868 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:56.148018 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.148514 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.149479 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.150224 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.150285 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.150295 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:56.150431 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 02:57:56.150469 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 02:57:56.150559 7720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.147595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45269 (local address 127.0.98.3:43651)
0924 02:57:56.147656 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:56.147669 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:56.147716 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:56.147879 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:56.147880 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:56.147882 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:56.147957 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.147959 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.148584 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.148585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.149398 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:56.149401 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.150293 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.150294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:56.150371 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.150373 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 02:57:56.150418 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 02:57:56.150454 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 02:57:56.150588 7397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ec6b81e4fe48839ba9d8fe6d6b79ac" candidate_uuid: "1631c5c9efc94c31975d48812ba9f320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cda0e259079447197b80ec4eb9acf9f" is_pre_election: true
I20260924 02:57:56.150743 7397 raft_consensus.cc:2454] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1631c5c9efc94c31975d48812ba9f320 in term 0.
I20260924 02:57:56.150872 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ec6b81e4fe48839ba9d8fe6d6b79ac" candidate_uuid: "1631c5c9efc94c31975d48812ba9f320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ada8ae64f647f8826a849c1377ac53" is_pre_election: true
I20260924 02:57:56.150951 7606 leader_election.cc:302] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1631c5c9efc94c31975d48812ba9f320, 6cda0e259079447197b80ec4eb9acf9f; no voters:
I20260924 02:57:56.150987 7534 raft_consensus.cc:2454] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1631c5c9efc94c31975d48812ba9f320 in term 0.
I20260924 02:57:56.151130 7750 raft_consensus.cc:2781] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:57:56.151178 7750 raft_consensus.cc:482] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:57:56.151201 7750 raft_consensus.cc:3037] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:56.151839 7750 raft_consensus.cc:504] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } }
I20260924 02:57:56.151964 7750 leader_election.cc:288] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [CANDIDATE]: Term 1 election: Requested vote from peers 50ada8ae64f647f8826a849c1377ac53 (127.0.98.2:45269), 6cda0e259079447197b80ec4eb9acf9f (127.0.98.1:35253)
I20260924 02:57:56.152101 7397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ec6b81e4fe48839ba9d8fe6d6b79ac" candidate_uuid: "1631c5c9efc94c31975d48812ba9f320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cda0e259079447197b80ec4eb9acf9f"
I20260924 02:57:56.152108 7534 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "68ec6b81e4fe48839ba9d8fe6d6b79ac" candidate_uuid: "1631c5c9efc94c31975d48812ba9f320" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "50ada8ae64f647f8826a849c1377ac53"
I20260924 02:57:56.152175 7397 raft_consensus.cc:3037] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:56.152182 7534 raft_consensus.cc:3037] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:56.152858 7534 raft_consensus.cc:2454] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1631c5c9efc94c31975d48812ba9f320 in term 1.
I20260924 02:57:56.152872 7397 raft_consensus.cc:2454] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1631c5c9efc94c31975d48812ba9f320 in term 1.
I20260924 02:57:56.153014 7606 leader_election.cc:302] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1631c5c9efc94c31975d48812ba9f320, 6cda0e259079447197b80ec4eb9acf9f; no voters:
I20260924 02:57:56.153101 7750 raft_consensus.cc:2781] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:56.153245 7750 raft_consensus.cc:686] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [term 1 LEADER]: Becoming Leader. State: Replica: 1631c5c9efc94c31975d48812ba9f320, State: Running, Role: LEADER
I20260924 02:57:56.153328 7750 consensus_queue.cc:237] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } }
I20260924 02:57:56.154389 6472 catalog_manager.cc:5495] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1631c5c9efc94c31975d48812ba9f320 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "1631c5c9efc94c31975d48812ba9f320" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } health_report { overall_health: HEALTHY } } }
I20260924 02:57:56.154960 7034 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: NON_VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: true } } }
I20260924 02:57:56.155055 7034 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:56.155576 7034 raft_consensus.cc:1050] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268: attempting to promote NON_VOTER 0cfa1c2eeee74f3e826cfe62bc1787ba to VOTER
I20260924 02:57:56.155704 7034 consensus_queue.cc:237] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 1.10, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false } }
I20260924 02:57:56.156059 7082 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 1 FOLLOWER]: Refusing update from remote peer eac45a5647564b07822c25ba28440268: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 02:57:56.156188 7274 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 1 LEARNER]: Refusing update from remote peer eac45a5647564b07822c25ba28440268: Log matching property violated. Preceding OpId in replica: term: 1 index: 10. Preceding OpId from leader: term: 1 index: 11. (index mismatch)
I20260924 02:57:56.156229 7034 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260924 02:57:56.156361 7034 consensus_queue.cc:1035] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 11, Last known committed idx: 9, Time since last communication: 0.000s
I20260924 02:57:56.156721 7758 raft_consensus.cc:2932] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [term 1 LEADER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 0cfa1c2eeee74f3e826cfe62bc1787ba (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false } } }
I20260924 02:57:56.156966 7082 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 0cfa1c2eeee74f3e826cfe62bc1787ba (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false } } }
I20260924 02:57:56.157634 7758 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false } } }
I20260924 02:57:56.157712 7758 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:56.157665 7034 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false } } }
I20260924 02:57:56.157760 7034 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:56.158005 7760 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false } } }
I20260924 02:57:56.158103 7760 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 02:57:56.158121 7034 sys_catalog.cc:455] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false } } }
I20260924 02:57:56.158196 7034 sys_catalog.cc:458] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:56.158382 7274 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 1 FOLLOWER]: Committing config change with OpId 1.11: config changed from index 7 to 11, 0cfa1c2eeee74f3e826cfe62bc1787ba (127.0.98.60) changed from NON_VOTER to VOTER. New config: { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false } } }
I20260924 02:57:56.159099 7303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "eac45a5647564b07822c25ba28440268" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false } } }
I20260924 02:57:56.159178 7303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: This master's current role is: FOLLOWER
I20260924 02:57:56.180081 7740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.177868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59862 (local address 127.0.98.3:43571)
0924 02:57:56.177919 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:56.177921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:56.177945 (+ 24us) server_negotiation.cc:407] Connection header received
0924 02:57:56.177977 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:56.177978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:56.177995 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:56.178044 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:56.178149 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.178742 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.179490 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.179671 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.179718 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.179828 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:56.179931 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:56.179985 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:56.179997 (+ 12us) server_negotiation.cc:299] Negotiation successful
0924 02:57:56.180012 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 02:57:56.181748 7397 raft_consensus.cc:1261] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f [term 1 FOLLOWER]: Refusing update from remote peer 1631c5c9efc94c31975d48812ba9f320: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:56.181773 7534 raft_consensus.cc:1261] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53 [term 1 FOLLOWER]: Refusing update from remote peer 1631c5c9efc94c31975d48812ba9f320: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:56.182073 7750 consensus_queue.cc:1035] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:56.182200 7755 consensus_queue.cc:1035] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [LEADER]: Connected to new peer: Peer: permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:56.185040 7763 mvcc.cc:205] Tried to move back new op lower bound from 7332735697638150144 to 7332735697524703232. Current Snapshot: MvccSnapshot[applied={T|T < 7332735697638150144}]
I20260924 02:57:56.187826 6472 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:56.187925 6472 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:56.189307 6472 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eac45a5647564b07822c25ba28440268: Sending DeleteTablet for 3 replicas of tablet 68ec6b81e4fe48839ba9d8fe6d6b79ac
I20260924 02:57:56.189647 7377 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ec6b81e4fe48839ba9d8fe6d6b79ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60238
I20260924 02:57:56.189690 7514 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ec6b81e4fe48839ba9d8fe6d6b79ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34094
I20260924 02:57:56.190511 7773 tablet_replica.cc:330] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53: stopping tablet replica
I20260924 02:57:56.190511 7650 tablet_service.cc:1514] Processing DeleteTablet for tablet 68ec6b81e4fe48839ba9d8fe6d6b79ac with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:56 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59850
I20260924 02:57:56.190665 7773 raft_consensus.cc:2229] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:56.190718 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 6441
I20260924 02:57:56.190771 7773 raft_consensus.cc:2258] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:56.191293 7773 ts_tablet_manager.cc:1905] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:56.191401 7772 tablet_replica.cc:330] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f: stopping tablet replica
I20260924 02:57:56.191494 7772 raft_consensus.cc:2229] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:56.191594 7772 raft_consensus.cc:2258] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:56.191920 7772 ts_tablet_manager.cc:1905] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:56.193055 7772 ts_tablet_manager.cc:1918] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:57:56.193113 7772 log.cc:1196] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-0/wal/wals/68ec6b81e4fe48839ba9d8fe6d6b79ac
I20260924 02:57:56.193325 7772 ts_tablet_manager.cc:1939] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 6cda0e259079447197b80ec4eb9acf9f: Deleting consensus metadata
I20260924 02:57:56.194636 7773 ts_tablet_manager.cc:1918] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:57:56.194710 7773 log.cc:1196] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/68ec6b81e4fe48839ba9d8fe6d6b79ac
I20260924 02:57:56.194926 7773 ts_tablet_manager.cc:1939] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 50ada8ae64f647f8826a849c1377ac53: Deleting consensus metadata
I20260924 02:57:56.195791 7775 tablet_replica.cc:330] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320: stopping tablet replica
I20260924 02:57:56.195909 7775 raft_consensus.cc:2229] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [term 1 LEADER]: Raft consensus shutting down.
I20260924 02:57:56.195994 7775 raft_consensus.cc:2258] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:56.196784 7775 ts_tablet_manager.cc:1905] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 02:57:56.197350 7469 connection.cc:535] server connection from 127.0.0.1:34094 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260924 02:57:56.197682 7332 connection.cc:535] server connection from 127.0.0.1:60238 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 02:57:56.199018 7775 ts_tablet_manager.cc:1918] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:57:56.199148 7775 log.cc:1196] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/68ec6b81e4fe48839ba9d8fe6d6b79ac
I20260924 02:57:56.199455 7775 ts_tablet_manager.cc:1939] T 68ec6b81e4fe48839ba9d8fe6d6b79ac P 1631c5c9efc94c31975d48812ba9f320: Deleting consensus metadata
W20260924 02:57:56.199801 7603 connection.cc:441] server connection from 127.0.0.1:59850 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59850 (request call id 1) could send its response
I20260924 02:57:56.202186 7731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.190066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40568 (local address 127.0.98.60:36985)
0924 02:57:56.190106 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:56.190108 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:56.190179 (+ 71us) server_negotiation.cc:407] Connection header received
0924 02:57:56.190258 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:56.190259 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:56.190270 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:56.190323 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:56.195719 (+ 5396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.196339 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.197282 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.197547 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.197602 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.199359 (+ 1757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:56.199361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:56.199361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:56.199368 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:56.200930 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.201274 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.201275 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.201276 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.201329 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.201494 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.201495 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.201495 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.201602 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:56.201682 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:56.202024 (+ 342us) server_negotiation.cc:299] Negotiation successful
0924 02:57:56.202097 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 02:57:56.206020 7732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.198914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38868 (local address 127.0.98.61:41113)
0924 02:57:56.198963 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:56.198965 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:56.199223 (+ 258us) server_negotiation.cc:407] Connection header received
0924 02:57:56.199324 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:56.199325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:56.199338 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:56.199367 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:56.199999 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.200730 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.201719 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.202006 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.202046 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.203394 (+ 1348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:56.203395 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:56.203395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:56.203402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:56.204835 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.205188 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.205190 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.205190 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.205247 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.205444 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.205445 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.205445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.205574 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:56.205642 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:56.205831 (+ 189us) server_negotiation.cc:299] Negotiation successful
0924 02:57:56.205925 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 02:57:56.206755 7449 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:56.200163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40582 (local address 127.0.98.60:36985)
0924 02:57:56.200199 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:56.200201 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:56.200322 (+ 121us) server_negotiation.cc:407] Connection header received
0924 02:57:56.200410 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:56.200411 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:56.200420 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:56.200446 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:56.201028 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.201755 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.202911 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:56.203201 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:56.203237 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:56.204501 (+ 1264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:56.204502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:56.204502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:56.204509 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:56.205735 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.206025 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.206026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.206026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.206075 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:56.206288 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:56.206289 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:56.206289 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:56.206402 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:56.206486 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:56.206640 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 02:57:56.206700 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 02:57:57.186475 7782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.186216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.2:32859)
0924 02:57:57.186342 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.186431 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
W20260924 02:57:57.186723 7581 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37993 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
I20260924 02:57:57.188078 7783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.187867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.1:45073)
0924 02:57:57.187976 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.188035 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":47,"threads_started":1}
W20260924 02:57:57.188264 7444 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37993 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
I20260924 02:57:57.188468 7784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.187922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.3:41715)
0924 02:57:57.188344 (+ 422us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.188425 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":407,"thread_start_us":53,"threads_started":1}
W20260924 02:57:57.188625 7717 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37993 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
I20260924 02:57:57.689625 7785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eac45a5647564b07822c25ba28440268)
I20260924 02:57:57.689718 7785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false } }
I20260924 02:57:57.689924 7785 leader_election.cc:288] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eac45a5647564b07822c25ba28440268 (127.0.98.62:37993), a8332978b13547c88de3e29f5b370cf3 (127.0.98.61:41113)
I20260924 02:57:57.690271 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.690044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.0.1:35114)
0924 02:57:57.690167 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.690229 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":47,"threads_started":1}
W20260924 02:57:57.690413 7251 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
I20260924 02:57:57.690279 7788 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader eac45a5647564b07822c25ba28440268)
I20260924 02:57:57.690567 7788 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false } }
I20260924 02:57:57.690825 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.690732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.0.1:35126)
0924 02:57:57.690769 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.690798 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260924 02:57:57.690920 7251 leader_election.cc:334] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer eac45a5647564b07822c25ba28440268 (127.0.98.62:37993): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
I20260924 02:57:57.690971 7788 leader_election.cc:288] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers eac45a5647564b07822c25ba28440268 (127.0.98.62:37993), 0cfa1c2eeee74f3e826cfe62bc1787ba (127.0.98.60:36985)
I20260924 02:57:57.691092 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.690876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.0.1:35128)
0924 02:57:57.690998 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.691055 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":40,"threads_started":1}
W20260924 02:57:57.691202 7059 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
I20260924 02:57:57.691619 7792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.691522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.0.1:35140)
0924 02:57:57.691561 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.691591 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 02:57:57.691699 7059 leader_election.cc:334] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer eac45a5647564b07822c25ba28440268 (127.0.98.62:37993): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
I20260924 02:57:57.693627 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.690160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38872 (local address 127.0.98.61:41113)
0924 02:57:57.690259 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:57.690261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:57.690272 (+ 11us) server_negotiation.cc:407] Connection header received
0924 02:57:57.690585 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:57.690587 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:57.690599 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:57.690767 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:57.690835 (+ 68us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.691650 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.692831 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.693338 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.693416 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:57.693428 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:57.693531 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 02:57:57.693567 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":44,"threads_started":1}
I20260924 02:57:57.693636 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.690105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:41113 (local address 127.0.0.1:38872)
0924 02:57:57.690216 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.690226 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:57.690282 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:57.690706 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:57.690707 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:57.690710 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:57.690770 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:57.690772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.691733 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.691734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:57.692738 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:57.692740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.693415 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.693416 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:57.693466 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:57.693469 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:57.693514 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 02:57:57.693588 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":39,"thread_start_us":46,"threads_started":1}
I20260924 02:57:57.693845 7082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a8332978b13547c88de3e29f5b370cf3" is_pre_election: true
I20260924 02:57:57.693924 7082 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cfa1c2eeee74f3e826cfe62bc1787ba in term 1.
I20260924 02:57:57.694056 7250 leader_election.cc:302] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfa1c2eeee74f3e826cfe62bc1787ba, a8332978b13547c88de3e29f5b370cf3; no voters: eac45a5647564b07822c25ba28440268
I20260924 02:57:57.694140 7785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260924 02:57:57.694177 7785 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 1 FOLLOWER]: Starting leader election (detected failure of leader eac45a5647564b07822c25ba28440268)
I20260924 02:57:57.694196 7785 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 1 FOLLOWER]: Advancing to term 2
I20260924 02:57:57.694272 7791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.691064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:36985 (local address 127.0.0.1:40588)
0924 02:57:57.691155 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.691164 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:57.691212 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:57.691541 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:57.691541 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:57.691545 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:57.691581 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:57.691582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.692226 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.692227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:57.693337 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:57.693339 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.694075 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.694076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:57.694134 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:57.694137 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:57.694179 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 02:57:57.694213 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":83,"thread_start_us":42,"threads_started":1}
I20260924 02:57:57.694311 7794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.691308 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40588 (local address 127.0.98.60:36985)
0924 02:57:57.691424 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:57.691426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:57.691434 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:57:57.691455 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:57.691456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:57.691465 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:57.691529 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:57.691642 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.692152 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.693469 (+ 1317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.694001 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.694077 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:57.694088 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:57.694211 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 02:57:57.694251 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":79,"threads_started":1}
I20260924 02:57:57.694463 7274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a8332978b13547c88de3e29f5b370cf3" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" is_pre_election: true
I20260924 02:57:57.695003 7785 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 2 FOLLOWER]: Starting leader election with config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false } }
I20260924 02:57:57.695114 7274 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a8332978b13547c88de3e29f5b370cf3 in current term 2: Already voted for candidate 0cfa1c2eeee74f3e826cfe62bc1787ba in this term.
I20260924 02:57:57.695125 7785 leader_election.cc:288] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [CANDIDATE]: Term 2 election: Requested vote from peers eac45a5647564b07822c25ba28440268 (127.0.98.62:37993), a8332978b13547c88de3e29f5b370cf3 (127.0.98.61:41113)
I20260924 02:57:57.695255 7082 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "a8332978b13547c88de3e29f5b370cf3"
I20260924 02:57:57.695331 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.695198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.0.1:35152)
0924 02:57:57.695288 (+ 90us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.695311 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260924 02:57:57.695307 7056 leader_election.cc:302] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a8332978b13547c88de3e29f5b370cf3; no voters: 0cfa1c2eeee74f3e826cfe62bc1787ba, eac45a5647564b07822c25ba28440268
I20260924 02:57:57.695319 7082 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 1 FOLLOWER]: Advancing to term 2
I20260924 02:57:57.695592 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.695511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.0.1:35168)
0924 02:57:57.695552 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.695576 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 02:57:57.695655 7251 leader_election.cc:334] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer eac45a5647564b07822c25ba28440268 (127.0.98.62:37993): Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
I20260924 02:57:57.695953 7082 raft_consensus.cc:2454] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cfa1c2eeee74f3e826cfe62bc1787ba in term 2.
I20260924 02:57:57.696022 7788 raft_consensus.cc:2726] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 02:57:57.696090 7250 leader_election.cc:302] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cfa1c2eeee74f3e826cfe62bc1787ba, a8332978b13547c88de3e29f5b370cf3; no voters: eac45a5647564b07822c25ba28440268
I20260924 02:57:57.696165 7785 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 2 FOLLOWER]: Leader election won for term 2
I20260924 02:57:57.696271 7785 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [term 2 LEADER]: Becoming Leader. State: Replica: 0cfa1c2eeee74f3e826cfe62bc1787ba, State: Running, Role: LEADER
I20260924 02:57:57.696327 7785 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false } }
I20260924 02:57:57.696623 7795 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cfa1c2eeee74f3e826cfe62bc1787ba. Latest consensus state: current_term: 2 leader_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false } } }
I20260924 02:57:57.696708 7795 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:57.696867 7797 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:57.697274 7797 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:57:57.697342 7797 catalog_manager.cc:1259] Loaded cluster ID: 58c36acd554e4645bd70b6b2991339b9
I20260924 02:57:57.697372 7797 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:57:57.697662 7797 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:57.697752 7797 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: Loaded TSK: 0
I20260924 02:57:57.697887 7797 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 02:57:57.733619 7264 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40582:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 02:57:57.735268 7264 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:57:57.736227 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.736124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.0.1:35184)
0924 02:57:57.736169 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.736200 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 02:57:57.736245 7082 raft_consensus.cc:1261] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [term 2 FOLLOWER]: Refusing update from remote peer 0cfa1c2eeee74f3e826cfe62bc1787ba: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260924 02:57:57.736446 7795 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [LEADER]: Connected to new peer: Peer: permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260924 02:57:57.736514 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.736447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.0.1:35200)
0924 02:57:57.736476 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.736498 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 02:57:57.736595 7251 consensus_peers.cc:487] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba -> Peer eac45a5647564b07822c25ba28440268 (127.0.98.62:37993): Couldn't send request to peer eac45a5647564b07822c25ba28440268. Status: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260924 02:57:57.737043 7788 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0cfa1c2eeee74f3e826cfe62bc1787ba. Latest consensus state: current_term: 2 leader_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false } } }
I20260924 02:57:57.737135 7788 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 02:57:57.737399 7798 mvcc.cc:205] Tried to move back new op lower bound from 7332735704005885952 to 7332735703844765696. Current Snapshot: MvccSnapshot[applied={T|T < 7332735704005885952}]
I20260924 02:57:57.737468 7785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false } } }
I20260924 02:57:57.737542 7785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:57.737663 7788 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" committed_config { opid_index: 11 OBSOLETE_local: true peers { permanent_uuid: "eac45a5647564b07822c25ba28440268" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37993 } } peers { permanent_uuid: "a8332978b13547c88de3e29f5b370cf3" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 41113 } attrs { promote: false } } peers { permanent_uuid: "0cfa1c2eeee74f3e826cfe62bc1787ba" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 36985 } attrs { promote: false } } }
I20260924 02:57:57.737753 7788 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8332978b13547c88de3e29f5b370cf3 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 02:57:57.737917 7799 mvcc.cc:205] Tried to move back new op lower bound from 7332735704005885952 to 7332735703844765696. Current Snapshot: MvccSnapshot[applied={T|T < 7332735704005885952 or (T in {7332735704005885952})}]
I20260924 02:57:57.742079 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.739124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43571 (local address 127.0.0.1:59874)
0924 02:57:57.739180 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.739191 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:57.739230 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:57.739473 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:57.739474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:57.739476 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:57.739527 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:57.739528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.740234 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.740235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:57.741173 (+ 938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:57.741175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.741859 (+ 684us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.741860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:57.741919 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:57.741923 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:57:57.741975 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 02:57:57.742010 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 02:57:57.742094 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.739232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59874 (local address 127.0.98.3:43571)
0924 02:57:57.739348 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:57.739350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:57.739357 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:57:57.739379 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:57.739380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:57.739391 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:57.739580 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:57.739614 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.740166 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.741260 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.741790 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.741855 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:57.741866 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:57.741986 (+ 120us) server_negotiation.cc:299] Negotiation successful
0924 02:57:57.742026 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":72,"threads_started":1}
I20260924 02:57:57.742384 7805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.739189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45269 (local address 127.0.0.1:34108)
0924 02:57:57.739449 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.739457 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:57.739504 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:57.739673 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:57.739674 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:57.739675 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:57.739722 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:57.739723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.740623 (+ 900us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.740624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:57.741464 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:57.741466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.742170 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.742171 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:57.742230 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:57.742234 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:57:57.742283 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 02:57:57.742319 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":160,"thread_start_us":71,"threads_started":1}
I20260924 02:57:57.742409 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.739253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34108 (local address 127.0.98.2:45269)
0924 02:57:57.739466 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:57.739468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:57.739499 (+ 31us) server_negotiation.cc:407] Connection header received
0924 02:57:57.739550 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:57.739571 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:57.739580 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:57.739634 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:57.739790 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.740529 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.741550 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.742067 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.742128 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:57.742139 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:57.742290 (+ 151us) server_negotiation.cc:299] Negotiation successful
0924 02:57:57.742331 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":64,"threads_started":1}
I20260924 02:57:57.742430 7650 tablet_service.cc:1467] Processing CreateTablet for tablet 6a120882230f4f838f5de4aa70c93244 (DEFAULT_TABLE table=test-table [id=1ca84c62975c4709aa3549499806a3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:57.742575 7650 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a120882230f4f838f5de4aa70c93244. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:57.742602 7514 tablet_service.cc:1467] Processing CreateTablet for tablet 6a120882230f4f838f5de4aa70c93244 (DEFAULT_TABLE table=test-table [id=1ca84c62975c4709aa3549499806a3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:57.742707 7514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a120882230f4f838f5de4aa70c93244. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:57.742761 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.739184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35253 (local address 127.0.0.1:60246)
0924 02:57:57.739279 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.739288 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:57.739336 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:57.739631 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:57.739631 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:57.739634 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:57.739678 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:57.739680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.740614 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.740615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:57.741632 (+ 1017us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:57.741634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.742571 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.742572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:57.742634 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:57.742637 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:57.742668 (+ 31us) client_negotiation.cc:240] Negotiation successful
0924 02:57:57.742705 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":24}
I20260924 02:57:57.743024 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.739411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60246 (local address 127.0.98.1:35253)
0924 02:57:57.739503 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:57.739505 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:57.739511 (+ 6us) server_negotiation.cc:407] Connection header received
0924 02:57:57.739534 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:57.739534 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:57.739543 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:57.739739 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:57.739784 (+ 45us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.740546 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.741710 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.742357 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.742915 (+ 558us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:57.742927 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:57.742942 (+ 15us) server_negotiation.cc:299] Negotiation successful
0924 02:57:57.742971 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":47,"threads_started":1}
I20260924 02:57:57.744091 7377 tablet_service.cc:1467] Processing CreateTablet for tablet 6a120882230f4f838f5de4aa70c93244 (DEFAULT_TABLE table=test-table [id=1ca84c62975c4709aa3549499806a3c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:57.744204 7377 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6a120882230f4f838f5de4aa70c93244. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:57.744541 7807 tablet_bootstrap.cc:492] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320: Bootstrap starting.
I20260924 02:57:57.744818 7807 tablet_bootstrap.cc:654] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:57.745038 7808 tablet_bootstrap.cc:492] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53: Bootstrap starting.
I20260924 02:57:57.745272 7807 tablet_bootstrap.cc:492] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320: No bootstrap required, opened a new log
I20260924 02:57:57.745323 7807 ts_tablet_manager.cc:1397] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 02:57:57.745388 7808 tablet_bootstrap.cc:654] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:57.745460 7810 tablet_bootstrap.cc:492] T 6a120882230f4f838f5de4aa70c93244 P 6cda0e259079447197b80ec4eb9acf9f: Bootstrap starting.
I20260924 02:57:57.745431 7807 raft_consensus.cc:348] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } }
I20260924 02:57:57.745504 7807 raft_consensus.cc:374] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:57.745532 7807 raft_consensus.cc:729] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1631c5c9efc94c31975d48812ba9f320, State: Initialized, Role: FOLLOWER
I20260924 02:57:57.745601 7807 consensus_queue.cc:260] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } }
I20260924 02:57:57.745743 7807 ts_tablet_manager.cc:1428] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:57.745803 7810 tablet_bootstrap.cc:654] T 6a120882230f4f838f5de4aa70c93244 P 6cda0e259079447197b80ec4eb9acf9f: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:57.746295 7716 heartbeater.cc:503] Master 127.0.98.60:36985 was elected leader, sending a full tablet report...
I20260924 02:57:57.746295 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.745930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.3:40519)
0924 02:57:57.746221 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.746261 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":247,"thread_start_us":69,"threads_started":1}
I20260924 02:57:57.746411 7808 tablet_bootstrap.cc:492] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53: No bootstrap required, opened a new log
I20260924 02:57:57.746447 7808 ts_tablet_manager.cc:1397] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:57.746601 7808 raft_consensus.cc:348] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } }
I20260924 02:57:57.746672 7808 raft_consensus.cc:374] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:57.746704 7808 raft_consensus.cc:729] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 50ada8ae64f647f8826a849c1377ac53, State: Initialized, Role: FOLLOWER
I20260924 02:57:57.746759 7808 consensus_queue.cc:260] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } }
I20260924 02:57:57.746786 7813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.746571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.3:53867)
0924 02:57:57.746735 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.746761 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":66,"threads_started":1}
I20260924 02:57:57.746917 7808 ts_tablet_manager.cc:1428] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260924 02:57:57.747360 7580 heartbeater.cc:503] Master 127.0.98.60:36985 was elected leader, sending a full tablet report...
I20260924 02:57:57.747347 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.747091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.2:38437)
0924 02:57:57.747276 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.747308 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167,"thread_start_us":67,"threads_started":1}
I20260924 02:57:57.747511 7810 tablet_bootstrap.cc:492] T 6a120882230f4f838f5de4aa70c93244 P 6cda0e259079447197b80ec4eb9acf9f: No bootstrap required, opened a new log
I20260924 02:57:57.747561 7810 ts_tablet_manager.cc:1397] T 6a120882230f4f838f5de4aa70c93244 P 6cda0e259079447197b80ec4eb9acf9f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:57.747751 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.747604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.2:50545)
0924 02:57:57.747708 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.747731 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":64,"threads_started":1}
I20260924 02:57:57.747709 7810 raft_consensus.cc:348] T 6a120882230f4f838f5de4aa70c93244 P 6cda0e259079447197b80ec4eb9acf9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } }
I20260924 02:57:57.747782 7810 raft_consensus.cc:374] T 6a120882230f4f838f5de4aa70c93244 P 6cda0e259079447197b80ec4eb9acf9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:57.747802 7810 raft_consensus.cc:729] T 6a120882230f4f838f5de4aa70c93244 P 6cda0e259079447197b80ec4eb9acf9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cda0e259079447197b80ec4eb9acf9f, State: Initialized, Role: FOLLOWER
I20260924 02:57:57.747920 7810 consensus_queue.cc:260] T 6a120882230f4f838f5de4aa70c93244 P 6cda0e259079447197b80ec4eb9acf9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } }
I20260924 02:57:57.748081 7810 ts_tablet_manager.cc:1428] T 6a120882230f4f838f5de4aa70c93244 P 6cda0e259079447197b80ec4eb9acf9f: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:57.748418 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.748198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.1:39171)
0924 02:57:57.748350 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.748381 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":67,"threads_started":1}
I20260924 02:57:57.748467 7443 heartbeater.cc:503] Master 127.0.98.60:36985 was elected leader, sending a full tablet report...
I20260924 02:57:57.748692 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.748621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.1:57159)
0924 02:57:57.748648 (+ 27us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.748671 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 02:57:57.751477 7719 tablet.cc:2369] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 02:57:57.844717 7816 raft_consensus.cc:482] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:57:57.844806 7816 raft_consensus.cc:504] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } }
I20260924 02:57:57.845151 7816 leader_election.cc:288] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1631c5c9efc94c31975d48812ba9f320 (127.0.98.3:43571), 6cda0e259079447197b80ec4eb9acf9f (127.0.98.1:35253)
I20260924 02:57:57.847898 7803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.845375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41025 (local address 127.0.98.3:43571)
0924 02:57:57.845540 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:57.845541 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:57.845548 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:57:57.845568 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:57.845569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:57.845578 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:57.845618 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:57.845757 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.846292 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.847236 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.847601 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.847658 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:57.847669 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:57.847787 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 02:57:57.847825 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260924 02:57:57.847899 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.845313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43571 (local address 127.0.98.2:41025)
0924 02:57:57.845415 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.845425 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:57.845471 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:57.845635 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:57.845636 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:57.845638 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:57.845695 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:57.845697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.846376 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.846377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:57.847162 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:57.847164 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.847668 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.847668 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:57.847729 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:57.847733 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:57:57.847776 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 02:57:57.847820 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76,"mutex_wait_us":37}
I20260924 02:57:57.847966 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.845386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40979 (local address 127.0.98.1:35253)
0924 02:57:57.845446 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:57.845448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:57.845458 (+ 10us) server_negotiation.cc:407] Connection header received
0924 02:57:57.845499 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:57.845500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:57.845512 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:57.845553 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:57.845681 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.846256 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.847278 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.847627 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.847708 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:57.847719 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:57.847861 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 02:57:57.847902 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 02:57:57.847983 7817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.845311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35253 (local address 127.0.98.2:40979)
0924 02:57:57.845373 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.845392 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:57.845445 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:57.845557 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:57.845557 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:57.845558 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:57.845627 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:57.845628 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.846326 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.846326 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:57.847200 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:57.847203 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.847726 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.847727 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:57.847791 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:57.847794 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:57.847849 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 02:57:57.847887 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":38}
I20260924 02:57:57.848311 7670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a120882230f4f838f5de4aa70c93244" candidate_uuid: "50ada8ae64f647f8826a849c1377ac53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1631c5c9efc94c31975d48812ba9f320" is_pre_election: true
I20260924 02:57:57.848383 7397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a120882230f4f838f5de4aa70c93244" candidate_uuid: "50ada8ae64f647f8826a849c1377ac53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cda0e259079447197b80ec4eb9acf9f" is_pre_election: true
I20260924 02:57:57.848443 7670 raft_consensus.cc:2454] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50ada8ae64f647f8826a849c1377ac53 in term 0.
I20260924 02:57:57.848459 7397 raft_consensus.cc:2454] T 6a120882230f4f838f5de4aa70c93244 P 6cda0e259079447197b80ec4eb9acf9f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 50ada8ae64f647f8826a849c1377ac53 in term 0.
I20260924 02:57:57.848592 7470 leader_election.cc:302] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 50ada8ae64f647f8826a849c1377ac53, 6cda0e259079447197b80ec4eb9acf9f; no voters:
I20260924 02:57:57.848711 7816 raft_consensus.cc:2781] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:57:57.848748 7816 raft_consensus.cc:482] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:57:57.848773 7816 raft_consensus.cc:3037] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:57.849435 7816 raft_consensus.cc:504] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } }
I20260924 02:57:57.849539 7816 leader_election.cc:288] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [CANDIDATE]: Term 1 election: Requested vote from peers 1631c5c9efc94c31975d48812ba9f320 (127.0.98.3:43571), 6cda0e259079447197b80ec4eb9acf9f (127.0.98.1:35253)
I20260924 02:57:57.849730 7397 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a120882230f4f838f5de4aa70c93244" candidate_uuid: "50ada8ae64f647f8826a849c1377ac53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cda0e259079447197b80ec4eb9acf9f"
I20260924 02:57:57.849722 7670 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6a120882230f4f838f5de4aa70c93244" candidate_uuid: "50ada8ae64f647f8826a849c1377ac53" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1631c5c9efc94c31975d48812ba9f320"
I20260924 02:57:57.849792 7397 raft_consensus.cc:3037] T 6a120882230f4f838f5de4aa70c93244 P 6cda0e259079447197b80ec4eb9acf9f [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:57.849797 7670 raft_consensus.cc:3037] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:57.850333 7397 raft_consensus.cc:2454] T 6a120882230f4f838f5de4aa70c93244 P 6cda0e259079447197b80ec4eb9acf9f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50ada8ae64f647f8826a849c1377ac53 in term 1.
I20260924 02:57:57.850471 7670 raft_consensus.cc:2454] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 50ada8ae64f647f8826a849c1377ac53 in term 1.
I20260924 02:57:57.850634 7468 leader_election.cc:302] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1631c5c9efc94c31975d48812ba9f320, 50ada8ae64f647f8826a849c1377ac53; no voters:
I20260924 02:57:57.850728 7816 raft_consensus.cc:2781] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:57.850880 7816 raft_consensus.cc:686] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [term 1 LEADER]: Becoming Leader. State: Replica: 50ada8ae64f647f8826a849c1377ac53, State: Running, Role: LEADER
I20260924 02:57:57.851007 7816 consensus_queue.cc:237] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } } peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } }
I20260924 02:57:57.851152 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.851062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.2:45363)
0924 02:57:57.851098 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.851125 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260924 02:57:57.851656 7263 catalog_manager.cc:5495] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 reported cstate change: term changed from 0 to 1, leader changed from <none> to 50ada8ae64f647f8826a849c1377ac53 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "50ada8ae64f647f8826a849c1377ac53" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "50ada8ae64f647f8826a849c1377ac53" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45269 } health_report { overall_health: HEALTHY } } }
I20260924 02:57:57.856444 7804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.854294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34112 (local address 127.0.98.2:45269)
0924 02:57:57.854349 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:57.854350 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:57.854358 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:57:57.854406 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:57.854407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:57.854424 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:57.854498 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:57.854615 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.855228 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.855899 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:57.856070 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:57.856110 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:57.856193 (+ 83us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:57.856290 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:57.856340 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:57.856370 (+ 30us) server_negotiation.cc:299] Negotiation successful
0924 02:57:57.856400 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 02:57:57.857276 7397 raft_consensus.cc:1261] T 6a120882230f4f838f5de4aa70c93244 P 6cda0e259079447197b80ec4eb9acf9f [term 1 FOLLOWER]: Refusing update from remote peer 50ada8ae64f647f8826a849c1377ac53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:57.857357 7670 raft_consensus.cc:1261] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320 [term 1 FOLLOWER]: Refusing update from remote peer 50ada8ae64f647f8826a849c1377ac53: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:57.857563 7816 consensus_queue.cc:1035] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cda0e259079447197b80ec4eb9acf9f" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 35253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:57.857684 7821 consensus_queue.cc:1035] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1631c5c9efc94c31975d48812ba9f320" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43571 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:57.858353 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.858232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.1:49211)
0924 02:57:57.858294 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.858323 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260924 02:57:57.858423 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.858311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.3:47805)
0924 02:57:57.858356 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.858395 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 02:57:57.859614 7820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.859534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.1:49395)
0924 02:57:57.859566 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.859588 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260924 02:57:57.859999 7818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.859904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.2:48129)
0924 02:57:57.859936 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.859970 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 02:57:57.860904 7263 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:57.860983 7263 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40582:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:57.860980 7815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:57.860902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37993 (local address 127.0.98.3:50407)
0924 02:57:57.860931 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.860956 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37993: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260924 02:57:57.859623 7825 mvcc.cc:205] Tried to move back new op lower bound from 7332735704501825536 to 7332735704478121984. Current Snapshot: MvccSnapshot[applied={T|T < 7332735704501825536}]
I20260924 02:57:57.862195 7263 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0cfa1c2eeee74f3e826cfe62bc1787ba: Sending DeleteTablet for 3 replicas of tablet 6a120882230f4f838f5de4aa70c93244
I20260924 02:57:57.862478 7650 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a120882230f4f838f5de4aa70c93244 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:57 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:59874
I20260924 02:57:57.862538 7514 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a120882230f4f838f5de4aa70c93244 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:57 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:34108
I20260924 02:57:57.862514 7377 tablet_service.cc:1514] Processing DeleteTablet for tablet 6a120882230f4f838f5de4aa70c93244 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:57 UTC) from {username='oryx', principal='oryx/127.0.98.60@KRBTEST.COM'} at 127.0.0.1:60246
I20260924 02:57:57.862727 7835 tablet_replica.cc:330] T 6a120882230f4f838f5de4aa70c93244 P 6cda0e259079447197b80ec4eb9acf9f: stopping tablet replica
I20260924 02:57:57.862779 7835 raft_consensus.cc:2229] T 6a120882230f4f838f5de4aa70c93244 P 6cda0e259079447197b80ec4eb9acf9f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:57.862818 7835 raft_consensus.cc:2258] T 6a120882230f4f838f5de4aa70c93244 P 6cda0e259079447197b80ec4eb9acf9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:57.863076 7833 tablet_replica.cc:330] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320: stopping tablet replica
I20260924 02:57:57.863123 7835 ts_tablet_manager.cc:1905] T 6a120882230f4f838f5de4aa70c93244 P 6cda0e259079447197b80ec4eb9acf9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:57.863184 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 7219
I20260924 02:57:57.863195 7833 raft_consensus.cc:2229] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:57.863294 7833 raft_consensus.cc:2258] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:57.863462 7834 tablet_replica.cc:330] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53: stopping tablet replica
I20260924 02:57:57.863539 7834 raft_consensus.cc:2229] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [term 1 LEADER]: Raft consensus shutting down.
I20260924 02:57:57.863641 7834 raft_consensus.cc:2258] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:57.863873 7834 ts_tablet_manager.cc:1905] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:57.864540 7833 ts_tablet_manager.cc:1905] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:57.865294 7834 ts_tablet_manager.cc:1918] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:57:57.865350 7834 log.cc:1196] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/6a120882230f4f838f5de4aa70c93244
I20260924 02:57:57.865585 7834 ts_tablet_manager.cc:1939] T 6a120882230f4f838f5de4aa70c93244 P 50ada8ae64f647f8826a849c1377ac53: Deleting consensus metadata
I20260924 02:57:57.865852 7248 catalog_manager.cc:4841] TS 50ada8ae64f647f8826a849c1377ac53 (127.0.98.2:45269): tablet 6a120882230f4f838f5de4aa70c93244 (table test-table [id=1ca84c62975c4709aa3549499806a3c7]) successfully deleted
I20260924 02:57:57.866285 7833 ts_tablet_manager.cc:1918] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:57:57.866333 7833 log.cc:1196] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/6a120882230f4f838f5de4aa70c93244
I20260924 02:57:57.866546 7833 ts_tablet_manager.cc:1939] T 6a120882230f4f838f5de4aa70c93244 P 1631c5c9efc94c31975d48812ba9f320: Deleting consensus metadata
I20260924 02:57:57.866858 7249 catalog_manager.cc:4841] TS 1631c5c9efc94c31975d48812ba9f320 (127.0.98.3:43571): tablet 6a120882230f4f838f5de4aa70c93244 (table test-table [id=1ca84c62975c4709aa3549499806a3c7]) successfully deleted
W20260924 02:57:57.871132 7786 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:57:57.870658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:35253 (local address 127.0.0.1:60248)
0924 02:57:57.870742 (+ 84us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:57.870752 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:57.870783 (+ 31us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:57.871087 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35253: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":67}
W20260924 02:57:57.871243 7251 catalog_manager.cc:4584] TS 6cda0e259079447197b80ec4eb9acf9f (127.0.98.1:35253): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6a120882230f4f838f5de4aa70c93244: Network error: Client connection negotiation failed: client connection to 127.0.98.1:35253: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 02:57:57.871613 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 7451
I20260924 02:57:57.876304 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 7587
I20260924 02:57:57.881302 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 6937
I20260924 02:57:57.885054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 7126
2026-09-24T02:57:57Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4446 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:57:57 dist-test-slave-sb6p krb5kdc[7838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:57:57 dist-test-slave-sb6p krb5kdc[7838](info): set up 2 sockets
Sep 24 02:57:57 dist-test-slave-sb6p krb5kdc[7838](info): commencing operation
krb5kdc: starting...
W20260924 02:57:58.830658 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:57:58 dist-test-slave-sb6p krb5kdc[7838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:57:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:57:58Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/oryx_127.0.98.62.keytab.
Entry for principal oryx/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/oryx_127.0.98.62.keytab.
I20260924 02:57:58.940912 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46449
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:59.020797 7851 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:59.021008 7851 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:59.021037 7851 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:59.022190 7851 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:57:59.022234 7851 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260924 02:57:59.022250 7851 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:59.022269 7851 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:59.022284 7851 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:57:59.022301 7851 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:57:59.023800 7851 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46449
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/oryx_127.0.98.62.keytab
--principal=oryx/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7851
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:59.024047 7851 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:59.024255 7851 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:59.028868 7857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:59.028942 7851 server_base.cc:1034] running on GCE node
W20260924 02:57:59.028895 7856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:59.028882 7859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:59.029263 7851 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:59.029524 7851 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:59.030654 7851 hybrid_clock.cc:648] HybridClock initialized: now 1790218679030646 us; error 26 us; skew 500 ppm
Sep 24 02:57:59 dist-test-slave-sb6p krb5kdc[7838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218679, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:59.033010 7851 init.cc:377] Logged in from keytab as oryx/127.0.98.62@KRBTEST.COM (short username oryx)
I20260924 02:57:59.033429 7851 webserver.cc:460] Webserver started at http://127.0.98.62:42813/ using document root <none> and password file <none>
I20260924 02:57:59.033634 7851 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:59.033680 7851 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:59.033798 7851 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:59.034823 7851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "6858ebb8fa404b13a1417e4b3e7c52b9"
format_stamp: "Formatted at 2026-09-24 02:57:59 on dist-test-slave-sb6p"
server_key: "a1fa15a25fede2767970dc3ac95fc3e6"
server_key_iv: "5ee13958185f56864f4a319677e73917"
server_key_version: "encryptionkey@0"
I20260924 02:57:59.035137 7851 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "6858ebb8fa404b13a1417e4b3e7c52b9"
format_stamp: "Formatted at 2026-09-24 02:57:59 on dist-test-slave-sb6p"
server_key: "a1fa15a25fede2767970dc3ac95fc3e6"
server_key_iv: "5ee13958185f56864f4a319677e73917"
server_key_version: "encryptionkey@0"
I20260924 02:57:59.036420 7851 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:57:59.037032 7866 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:59.037191 7851 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:59.037272 7851 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "6858ebb8fa404b13a1417e4b3e7c52b9"
format_stamp: "Formatted at 2026-09-24 02:57:59 on dist-test-slave-sb6p"
server_key: "a1fa15a25fede2767970dc3ac95fc3e6"
server_key_iv: "5ee13958185f56864f4a319677e73917"
server_key_version: "encryptionkey@0"
I20260924 02:57:59.037328 7851 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:59.049166 7851 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:59.051206 7851 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:59.051337 7851 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:59.055586 7851 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46449
I20260924 02:57:59.055640 7928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46449 every 8 connection(s)
I20260924 02:57:59.055941 7851 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:57:59.056509 7929 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:59.058655 7929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9: Bootstrap starting.
I20260924 02:57:59.059175 7929 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:59.059433 7929 log.cc:824] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:59.059996 7929 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9: No bootstrap required, opened a new log
I20260924 02:57:59.061985 7929 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6858ebb8fa404b13a1417e4b3e7c52b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46449 } }
I20260924 02:57:59.062120 7929 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:59.062151 7929 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6858ebb8fa404b13a1417e4b3e7c52b9, State: Initialized, Role: FOLLOWER
I20260924 02:57:59.062285 7929 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6858ebb8fa404b13a1417e4b3e7c52b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46449 } }
I20260924 02:57:59.062340 7929 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:57:59.062378 7929 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:57:59.062424 7929 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:59.062952 7929 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6858ebb8fa404b13a1417e4b3e7c52b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46449 } }
I20260924 02:57:59.063041 7929 leader_election.cc:302] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6858ebb8fa404b13a1417e4b3e7c52b9; no voters:
I20260924 02:57:59.063190 7929 leader_election.cc:288] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:57:59.063265 7932 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:59.063426 7932 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [term 1 LEADER]: Becoming Leader. State: Replica: 6858ebb8fa404b13a1417e4b3e7c52b9, State: Running, Role: LEADER
I20260924 02:57:59.063452 7929 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:57:59.063512 7932 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6858ebb8fa404b13a1417e4b3e7c52b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46449 } }
I20260924 02:57:59.063891 7933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6858ebb8fa404b13a1417e4b3e7c52b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6858ebb8fa404b13a1417e4b3e7c52b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46449 } } }
I20260924 02:57:59.063998 7933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:59.063863 7934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6858ebb8fa404b13a1417e4b3e7c52b9. Latest consensus state: current_term: 1 leader_uuid: "6858ebb8fa404b13a1417e4b3e7c52b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6858ebb8fa404b13a1417e4b3e7c52b9" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46449 } } }
I20260924 02:57:59.064163 7934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9 [sys.catalog]: This master's current role is: LEADER
I20260924 02:57:59.064179 7941 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:57:59.064669 7941 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:57:59.065038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 7851
I20260924 02:57:59.065135 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:57:59.065323 392 external_mini_cluster.cc:1428] Setting key 8bd03f8875c7c85c535af610e375e9cc
I20260924 02:57:59.066407 7941 catalog_manager.cc:1347] Generated new cluster ID: ba0a34b267284d2bb78be02c298eb393
I20260924 02:57:59.066445 7941 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 02:57:59 dist-test-slave-sb6p krb5kdc[7838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218678, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 02:57:59.076857 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.065768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53466 (local address 127.0.98.62:46449)
0924 02:57:59.066626 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.066635 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.066699 (+ 64us) server_negotiation.cc:407] Connection header received
0924 02:57:59.066837 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.066844 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.066898 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.067034 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:59.067747 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.069115 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.070163 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.070769 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.070862 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.073449 (+ 2587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:59.073454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:59.073454 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:59.073466 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:59.075338 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.075691 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.075694 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.075694 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.075797 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.075962 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.075962 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.075963 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.076255 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:59.076395 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.076479 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.076614 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":814,"thread_start_us":71,"threads_started":1}
I20260924 02:57:59.103008 7941 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:57:59.103542 7941 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:57:59.115190 7941 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9: Generated new TSK 0
I20260924 02:57:59.115365 7941 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/oryx_127.0.98.1.keytab.
I20260924 02:57:59.145402 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46449
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:59.224046 7955 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:59.224310 7955 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:59.224335 7955 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:59.225575 7955 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:59.225616 7955 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:59.225636 7955 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 02:57:59.225653 7955 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:57:59.227240 7955 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7955
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:59.227466 7955 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:59.227679 7955 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:59.232528 7961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:59.232563 7960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:59.232705 7955 server_base.cc:1034] running on GCE node
W20260924 02:57:59.232549 7963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:59.232947 7955 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:59.233171 7955 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:59.234303 7955 hybrid_clock.cc:648] HybridClock initialized: now 1790218679234295 us; error 27 us; skew 500 ppm
Sep 24 02:57:59 dist-test-slave-sb6p krb5kdc[7838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218679, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:59.236577 7955 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 02:57:59.237007 7955 webserver.cc:460] Webserver started at http://127.0.98.1:46509/ using document root <none> and password file <none>
I20260924 02:57:59.237214 7955 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:59.237260 7955 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:59.237377 7955 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:59.238407 7955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "71b2ecc8215f4e608fd14e6f07625b7a"
format_stamp: "Formatted at 2026-09-24 02:57:59 on dist-test-slave-sb6p"
server_key: "1cb6340e7f473cf7f03f28caddf0f137"
server_key_iv: "bea95e9f0a2617a73639a77624bed9b1"
server_key_version: "encryptionkey@0"
I20260924 02:57:59.238763 7955 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "71b2ecc8215f4e608fd14e6f07625b7a"
format_stamp: "Formatted at 2026-09-24 02:57:59 on dist-test-slave-sb6p"
server_key: "1cb6340e7f473cf7f03f28caddf0f137"
server_key_iv: "bea95e9f0a2617a73639a77624bed9b1"
server_key_version: "encryptionkey@0"
I20260924 02:57:59.240007 7955 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:59.240684 7970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:59.240840 7955 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:59.240942 7955 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "71b2ecc8215f4e608fd14e6f07625b7a"
format_stamp: "Formatted at 2026-09-24 02:57:59 on dist-test-slave-sb6p"
server_key: "1cb6340e7f473cf7f03f28caddf0f137"
server_key_iv: "bea95e9f0a2617a73639a77624bed9b1"
server_key_version: "encryptionkey@0"
I20260924 02:57:59.241000 7955 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:59.258064 7955 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:59.259750 7955 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:59.259887 7955 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:59.260344 7955 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:59.260386 7955 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:59.260416 7955 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:59.260433 7955 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:59.266830 7955 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37615
Sep 24 02:57:59 dist-test-slave-sb6p krb5kdc[7838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218679, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 02:57:59.266953 8085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37615 every 8 connection(s)
I20260924 02:57:59.267187 7955 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:57:59.269794 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 7955
I20260924 02:57:59.269886 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:57:59.269901 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.260797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56671 (local address 127.0.98.62:46449)
0924 02:57:59.260836 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.260838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.261055 (+ 217us) server_negotiation.cc:407] Connection header received
0924 02:57:59.261563 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.261564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.261577 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.261627 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:59.262454 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.263237 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.264669 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.264889 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.264947 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.267373 (+ 2426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:59.267382 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:59.267382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:59.267390 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:59.268642 (+ 1252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.269025 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.269026 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.269026 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.269076 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.269285 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.269286 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.269286 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.269398 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:59.269453 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.269727 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.269809 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 02:57:59.270059 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.260797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46449 (local address 127.0.98.1:56671)
0924 02:57:59.260945 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:59.260980 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:59.261467 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:59.261708 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:59.261712 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:59.262041 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:59.262314 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.262320 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.263326 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.263328 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.264586 (+ 1258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.264589 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.264962 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.264963 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.265110 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.265598 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:59.265608 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:59.267289 (+ 1681us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:59.268716 (+ 1427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:59.268723 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:59.268725 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:59.268956 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:59.269128 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:59.269129 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:59.269129 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:59.269231 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:59.269451 (+ 220us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:59.269457 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:59.269588 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 02:57:59.269697 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 02:57:59.269848 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260924 02:57:59.270130 392 external_mini_cluster.cc:1428] Setting key 369c1e24556d16ddda1502e0f7dadb1d
I20260924 02:57:59.270503 8086 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46449
I20260924 02:57:59.270605 8086 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:59.270772 8086 heartbeater.cc:511] Master 127.0.98.62:46449 requested a full tablet report, sending...
I20260924 02:57:59.271251 7882 ts_manager.cc:194] Registered new tserver with Master: 71b2ecc8215f4e608fd14e6f07625b7a (127.0.98.1:37615)
I20260924 02:57:59.272091 7882 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:56671
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 02:57:59.279565 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.271512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39417 (local address 127.0.98.62:46449)
0924 02:57:59.271546 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.271547 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.271633 (+ 86us) server_negotiation.cc:407] Connection header received
0924 02:57:59.271677 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.271677 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.271686 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.271741 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:59.272051 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.272733 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.273441 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.273697 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.273761 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.277549 (+ 3788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:59.277551 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:59.277551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:59.277558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:59.278682 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.278980 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.278980 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.278981 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.279025 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.279167 (+ 142us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.279168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.279168 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.279283 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:59.279340 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.279443 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.279504 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 02:57:59.279569 8092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.271453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46449 (local address 127.0.98.1:39417)
0924 02:57:59.271572 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:59.271582 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:59.271633 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:59.271742 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:59.271743 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:59.271914 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:59.271973 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.271974 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.272831 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.272832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.273364 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.273366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.273770 (+ 404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.273771 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.273820 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.274181 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:59.274185 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:59.274485 (+ 300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:59.278759 (+ 4274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:59.278760 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:59.278760 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:59.278929 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:59.279060 (+ 131us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:59.279060 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:59.279060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:59.279102 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:59.279332 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:59.279333 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:59.279367 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 02:57:59.279421 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 02:57:59.279496 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/oryx_127.0.98.2.keytab.
I20260924 02:57:59.293118 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46449
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:59.369642 8095 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:59.369841 8095 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:59.369871 8095 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:59.371054 8095 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:59.371094 8095 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:59.371120 8095 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 02:57:59.371142 8095 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:57:59.372625 8095 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8095
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:59.372852 8095 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:59.373054 8095 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:59.377712 8103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:59.377755 8100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:59.377738 8101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:59.377918 8095 server_base.cc:1034] running on GCE node
I20260924 02:57:59.378077 8095 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:59.378263 8095 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:59.379407 8095 hybrid_clock.cc:648] HybridClock initialized: now 1790218679379392 us; error 31 us; skew 500 ppm
Sep 24 02:57:59 dist-test-slave-sb6p krb5kdc[7838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218679, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:59.381584 8095 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 02:57:59.381985 8095 webserver.cc:460] Webserver started at http://127.0.98.2:38677/ using document root <none> and password file <none>
I20260924 02:57:59.382172 8095 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:59.382218 8095 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:59.382335 8095 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:59.383424 8095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "b738799e612c4554a4ad08102c417de0"
format_stamp: "Formatted at 2026-09-24 02:57:59 on dist-test-slave-sb6p"
server_key: "7cee393797c0b8e632b4355790ccf07e"
server_key_iv: "621533a9856dcd6c65b4473323904a3e"
server_key_version: "encryptionkey@0"
I20260924 02:57:59.383751 8095 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b738799e612c4554a4ad08102c417de0"
format_stamp: "Formatted at 2026-09-24 02:57:59 on dist-test-slave-sb6p"
server_key: "7cee393797c0b8e632b4355790ccf07e"
server_key_iv: "621533a9856dcd6c65b4473323904a3e"
server_key_version: "encryptionkey@0"
I20260924 02:57:59.384933 8095 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:57:59.385560 8110 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:59.385706 8095 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:59.385789 8095 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "b738799e612c4554a4ad08102c417de0"
format_stamp: "Formatted at 2026-09-24 02:57:59 on dist-test-slave-sb6p"
server_key: "7cee393797c0b8e632b4355790ccf07e"
server_key_iv: "621533a9856dcd6c65b4473323904a3e"
server_key_version: "encryptionkey@0"
I20260924 02:57:59.385851 8095 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:59.406509 8095 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:59.408347 8095 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:59.408468 8095 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:59.408913 8095 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:59.408950 8095 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:59.408978 8095 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:59.408995 8095 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 02:57:59 dist-test-slave-sb6p krb5kdc[7838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218679, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 02:57:59.415508 8095 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46053
I20260924 02:57:59.415834 8095 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 02:57:59.416258 8225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46053 every 8 connection(s)
I20260924 02:57:59.417177 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 8095
I20260924 02:57:59.417302 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:57:59.417502 392 external_mini_cluster.cc:1428] Setting key 56c4131dbdea92cc189e1f7dbae6da54
I20260924 02:57:59.418680 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.409321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53369 (local address 127.0.98.62:46449)
0924 02:57:59.409360 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.409362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.409567 (+ 205us) server_negotiation.cc:407] Connection header received
0924 02:57:59.410055 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.410057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.410077 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.410149 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:59.410874 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.411717 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.413207 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.413514 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.413582 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.415925 (+ 2343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:59.415926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:59.415927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:59.415935 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:59.417215 (+ 1280us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.417615 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.417616 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.417617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.417680 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.417960 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.417961 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.417961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.418092 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:59.418155 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.418549 (+ 394us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.418616 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 02:57:59.418895 8119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.409323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46449 (local address 127.0.98.2:53369)
0924 02:57:59.409464 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:59.409497 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:59.409960 (+ 463us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:59.410206 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:59.410209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:59.410509 (+ 300us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:59.410780 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.410785 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.411816 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.411817 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.413116 (+ 1299us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.413120 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.413602 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.413603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.413742 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.414223 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:59.414233 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:59.415839 (+ 1606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:59.417291 (+ 1452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:59.417297 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:59.417299 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:59.417530 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:59.417754 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:59.417755 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:59.417755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:59.417894 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:59.418151 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:59.418156 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:59.418388 (+ 232us) client_negotiation.cc:769] Sending connection context
0924 02:57:59.418497 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 02:57:59.418684 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":85,"threads_started":1}
I20260924 02:57:59.419389 8226 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46449
I20260924 02:57:59.419476 8226 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:59.419665 8226 heartbeater.cc:511] Master 127.0.98.62:46449 requested a full tablet report, sending...
I20260924 02:57:59.420001 7883 ts_manager.cc:194] Registered new tserver with Master: b738799e612c4554a4ad08102c417de0 (127.0.98.2:46053)
I20260924 02:57:59.420464 7883 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53369
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 02:57:59.426067 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.420377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43817 (local address 127.0.98.62:46449)
0924 02:57:59.420410 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.420412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.420556 (+ 144us) server_negotiation.cc:407] Connection header received
0924 02:57:59.420602 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.420603 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.420613 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.420669 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:59.421032 (+ 363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.421716 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.422713 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.422900 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.422957 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.423824 (+ 867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:59.423825 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:59.423826 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:59.423831 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:59.425015 (+ 1184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.425354 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.425354 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.425355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.425396 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.425536 (+ 140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.425536 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.425536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.425692 (+ 156us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:59.425772 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.425909 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.425975 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 02:57:59.426080 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.420327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46449 (local address 127.0.98.2:43817)
0924 02:57:59.420482 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:59.420493 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:59.420552 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:59.420709 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:59.420710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:59.420890 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:59.420954 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.420955 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.421828 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.421829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.422638 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.422641 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.422973 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.422974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.423038 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.423399 (+ 361us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:59.423403 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:59.423750 (+ 347us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:59.425093 (+ 1343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:59.425094 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:59.425094 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:59.425286 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:59.425433 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:59.425434 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:59.425435 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:59.425489 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:59.425776 (+ 287us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:59.425777 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:59.425820 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 02:57:59.425879 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 02:57:59.425976 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 02:57:59.440840 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:46449
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35083
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:57:59.517617 8235 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:57:59.517815 8235 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:57:59.517845 8235 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:57:59.519104 8235 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:57:59.519151 8235 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:57:59.519181 8235 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260924 02:57:59.519210 8235 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:57:59.520746 8235 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35083
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.98.62:46449
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8235
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:57:59.520987 8235 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:57:59.521186 8235 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:57:59.525782 8243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:57:59.525857 8241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:59.525921 8235 server_base.cc:1034] running on GCE node
W20260924 02:57:59.525785 8240 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:57:59.526188 8235 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:57:59.526439 8235 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:57:59.527556 8235 hybrid_clock.cc:648] HybridClock initialized: now 1790218679527539 us; error 29 us; skew 500 ppm
Sep 24 02:57:59 dist-test-slave-sb6p krb5kdc[7838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218679, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:57:59.529855 8235 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 02:57:59.530282 8235 webserver.cc:460] Webserver started at http://127.0.98.3:45727/ using document root <none> and password file <none>
I20260924 02:57:59.530478 8235 fs_manager.cc:359] Metadata directory not provided
I20260924 02:57:59.530546 8235 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:57:59.530673 8235 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:57:59.531706 8235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "47faee667646449abd37257851cb71a1"
format_stamp: "Formatted at 2026-09-24 02:57:59 on dist-test-slave-sb6p"
server_key: "9c84ad839b7f84456a7f01189e2f266e"
server_key_iv: "367af9790a720447723d17ecf04b1371"
server_key_version: "encryptionkey@0"
I20260924 02:57:59.532037 8235 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "47faee667646449abd37257851cb71a1"
format_stamp: "Formatted at 2026-09-24 02:57:59 on dist-test-slave-sb6p"
server_key: "9c84ad839b7f84456a7f01189e2f266e"
server_key_iv: "367af9790a720447723d17ecf04b1371"
server_key_version: "encryptionkey@0"
I20260924 02:57:59.533239 8235 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:57:59.533835 8250 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:59.533969 8235 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:59.534046 8235 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "47faee667646449abd37257851cb71a1"
format_stamp: "Formatted at 2026-09-24 02:57:59 on dist-test-slave-sb6p"
server_key: "9c84ad839b7f84456a7f01189e2f266e"
server_key_iv: "367af9790a720447723d17ecf04b1371"
server_key_version: "encryptionkey@0"
I20260924 02:57:59.534109 8235 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:57:59.545475 8235 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:57:59.547353 8235 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:57:59.547484 8235 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:57:59.547983 8235 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:57:59.548022 8235 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:59.548050 8235 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:57:59.548069 8235 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 24 02:57:59 dist-test-slave-sb6p krb5kdc[7838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218679, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 02:57:59.554461 8235 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:46189
I20260924 02:57:59.554538 8365 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:46189 every 8 connection(s)
I20260924 02:57:59.554900 8235 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 02:57:59.556820 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.548344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:32771 (local address 127.0.98.62:46449)
0924 02:57:59.548383 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.548385 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.548633 (+ 248us) server_negotiation.cc:407] Connection header received
0924 02:57:59.549150 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.549152 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.549164 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.549213 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:59.549931 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.550602 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.551558 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.551763 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.551817 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.554430 (+ 2613us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:59.554431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:59.554431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:59.554439 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:59.555776 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.556093 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.556093 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.556094 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.556137 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.556359 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.556360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.556360 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.556468 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:59.556512 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.556708 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.556759 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 02:57:59.556984 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.548352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46449 (local address 127.0.98.3:32771)
0924 02:57:59.548502 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:59.548555 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:59.549056 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:59.549283 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:59.549287 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:59.549600 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:59.549845 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.549852 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.550682 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.550683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.551480 (+ 797us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.551482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.551826 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.551827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.551951 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.552371 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:59.552380 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:59.554335 (+ 1955us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:59.555839 (+ 1504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:59.555842 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:59.555844 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:59.556035 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:59.556201 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:59.556202 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:59.556203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:59.556306 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:59.556511 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:59.556514 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:59.556621 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 02:57:59.556695 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 02:57:59.556806 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":83,"threads_started":1}
I20260924 02:57:59.557443 8366 heartbeater.cc:347] Connected to a master server at 127.0.98.62:46449
I20260924 02:57:59.557516 8366 heartbeater.cc:464] Registering TS with master...
I20260924 02:57:59.557663 8366 heartbeater.cc:511] Master 127.0.98.62:46449 requested a full tablet report, sending...
I20260924 02:57:59.558002 7882 ts_manager.cc:194] Registered new tserver with Master: 47faee667646449abd37257851cb71a1 (127.0.98.3:46189)
I20260924 02:57:59.558461 7882 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:32771
I20260924 02:57:59.562700 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.558337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45555 (local address 127.0.98.62:46449)
0924 02:57:59.558373 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.558374 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.558438 (+ 64us) server_negotiation.cc:407] Connection header received
0924 02:57:59.558478 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.558479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.558488 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.558562 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:59.558849 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.559444 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.560085 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.560255 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.560301 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.561004 (+ 703us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:59.561005 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:59.561006 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:59.561011 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:59.561952 (+ 941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.562203 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.562204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.562204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.562243 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.562349 (+ 106us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.562349 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.562349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.562430 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:59.562470 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.562591 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.562645 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 02:57:59.562712 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.558273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46449 (local address 127.0.98.3:45555)
0924 02:57:59.558388 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:59.558398 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:59.558445 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:59.558579 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:59.558580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:59.558744 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:59.558789 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.558790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.559517 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.559518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.560029 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.560031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.560305 (+ 274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.560306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.560347 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.560633 (+ 286us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:59.560636 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:59.560943 (+ 307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:59.562014 (+ 1071us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:59.562015 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:59.562015 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:59.562153 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:59.562277 (+ 124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:59.562278 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:59.562278 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:59.562317 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:59.562463 (+ 146us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:59.562463 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:59.562494 (+ 31us) client_negotiation.cc:769] Sending connection context
0924 02:57:59.562573 (+ 79us) client_negotiation.cc:240] Negotiation successful
0924 02:57:59.562648 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":65,"threads_started":1}
I20260924 02:57:59.564800 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 8235
I20260924 02:57:59.564888 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:57:59.565079 392 external_mini_cluster.cc:1428] Setting key b6ae87a9b155ae6f40552b32b4050c44
I20260924 02:57:59.565785 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 24 02:57:59 dist-test-slave-sb6p krb5kdc[7838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218679, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 24 02:57:59 dist-test-slave-sb6p krb5kdc[7838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:57:59 dist-test-slave-sb6p krb5kdc[7838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 02:57:59.584263 7951 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:57:59.579539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53468 (local address 127.0.98.62:46449)
0924 02:57:59.579590 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.579592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.579600 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:57:59.579665 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.579666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.579676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.579721 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:59.580219 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.580697 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.581396 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.581571 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.581626 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.584191 (+ 2565us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53468: BlockingRecv error: recv got EOF from 127.0.0.1:53468 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 02:57:59 dist-test-slave-sb6p krb5kdc[7838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218679, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 02:57:59.590798 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.585027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53476 (local address 127.0.98.62:46449)
0924 02:57:59.585082 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.585084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.585145 (+ 61us) server_negotiation.cc:407] Connection header received
0924 02:57:59.585206 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.585207 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.585216 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.585260 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:59.585660 (+ 400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.586113 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.586786 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.586976 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.587026 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.588845 (+ 1819us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:59.588847 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:59.588847 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:59.588854 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:59.589872 (+ 1018us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.590146 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.590147 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.590148 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.590196 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.590350 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.590351 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.590351 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.590455 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:59.590556 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.590647 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.590713 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 02:57:59.592103 7882 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 02:57:59.593619 7882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:57:59.599234 8386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.595409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37615 (local address 127.0.0.1:55534)
0924 02:57:59.595679 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:59.595717 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:59.595784 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:59.595948 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:59.595952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:59.595954 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:59.596010 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.596012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.596966 (+ 954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.596967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.597947 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.597950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.598939 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.598940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.599029 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.599043 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 02:57:59.599115 (+ 72us) client_negotiation.cc:240] Negotiation successful
0924 02:57:59.599161 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":62,"threads_started":1}
I20260924 02:57:59.599264 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.595533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55534 (local address 127.0.98.1:37615)
0924 02:57:59.595700 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.595705 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.595774 (+ 69us) server_negotiation.cc:407] Connection header received
0924 02:57:59.595845 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.595848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.595862 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.595934 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:59.596071 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.596896 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.598044 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.598856 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.598930 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.599001 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.599134 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.599172 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":68,"threads_started":1}
I20260924 02:57:59.599598 8385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.595427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46053 (local address 127.0.0.1:51970)
0924 02:57:59.595685 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:59.595721 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:59.595774 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:59.596090 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:59.596091 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:59.596093 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:59.596140 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.596142 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.597513 (+ 1371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.597514 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.598457 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.598460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.599416 (+ 956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.599417 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.599467 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.599470 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:59.599522 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 02:57:59.599551 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":52,"threads_started":1}
I20260924 02:57:59.599695 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.595522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51970 (local address 127.0.98.2:46053)
0924 02:57:59.595958 (+ 436us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.595962 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.595971 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:57:59.596006 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.596008 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.596022 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.596086 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:59.596194 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.597429 (+ 1235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.598550 (+ 1121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.599239 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.599306 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.599383 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.599561 (+ 178us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.599605 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":71,"threads_started":1}
I20260924 02:57:59.600090 8020 tablet_service.cc:1467] Processing CreateTablet for tablet ae932bce29794bc991b31770af10ae3a (DEFAULT_TABLE table=test-table [id=27b2bd130ffb49e391c09f9e8fe5508d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:59.600235 8160 tablet_service.cc:1467] Processing CreateTablet for tablet ae932bce29794bc991b31770af10ae3a (DEFAULT_TABLE table=test-table [id=27b2bd130ffb49e391c09f9e8fe5508d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:59.600540 8020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae932bce29794bc991b31770af10ae3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:59.600540 8160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae932bce29794bc991b31770af10ae3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:59.601018 8389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.595604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46189 (local address 127.0.0.1:34628)
0924 02:57:59.595858 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:59.595866 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:59.595918 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:59.596269 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:59.596270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:59.596272 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:59.596306 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.596308 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.598543 (+ 2235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.598544 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.600069 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.600072 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.600804 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.600805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.600863 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.600867 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:57:59.600915 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 02:57:59.600952 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":37,"threads_started":1}
I20260924 02:57:59.601050 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.595705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34628 (local address 127.0.98.3:46189)
0924 02:57:59.595865 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.595870 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.596083 (+ 213us) server_negotiation.cc:407] Connection header received
0924 02:57:59.596125 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.596127 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.596141 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.596207 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:59.596366 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.597439 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.600147 (+ 2708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.600726 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.600794 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.600869 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.600926 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.600960 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20260924 02:57:59.604276 8392 tablet_bootstrap.cc:492] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0: Bootstrap starting.
I20260924 02:57:59.604807 8392 tablet_bootstrap.cc:654] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:59.604748 8300 tablet_service.cc:1467] Processing CreateTablet for tablet ae932bce29794bc991b31770af10ae3a (DEFAULT_TABLE table=test-table [id=27b2bd130ffb49e391c09f9e8fe5508d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:59.605013 8300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ae932bce29794bc991b31770af10ae3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:59.605089 8392 log.cc:824] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:59.605389 8391 tablet_bootstrap.cc:492] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a: Bootstrap starting.
I20260924 02:57:59.605866 8392 tablet_bootstrap.cc:492] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0: No bootstrap required, opened a new log
I20260924 02:57:59.605945 8392 ts_tablet_manager.cc:1397] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:59.605957 8391 tablet_bootstrap.cc:654] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:59.606266 8391 log.cc:824] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:59.607031 8391 tablet_bootstrap.cc:492] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a: No bootstrap required, opened a new log
I20260924 02:57:59.607098 8391 ts_tablet_manager.cc:1397] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:59.607848 8395 tablet_bootstrap.cc:492] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1: Bootstrap starting.
I20260924 02:57:59.607951 8392 raft_consensus.cc:348] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } }
I20260924 02:57:59.608119 8392 raft_consensus.cc:374] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:59.608155 8392 raft_consensus.cc:729] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b738799e612c4554a4ad08102c417de0, State: Initialized, Role: FOLLOWER
I20260924 02:57:59.608289 8392 consensus_queue.cc:260] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } }
I20260924 02:57:59.608400 8395 tablet_bootstrap.cc:654] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:59.608546 8392 ts_tablet_manager.cc:1428] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 02:57:59.608661 8226 heartbeater.cc:503] Master 127.0.98.62:46449 was elected leader, sending a full tablet report...
I20260924 02:57:59.608700 8395 log.cc:824] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1: Log is configured to *not* fsync() on all Append() calls
I20260924 02:57:59.609315 8395 tablet_bootstrap.cc:492] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1: No bootstrap required, opened a new log
I20260924 02:57:59.609372 8395 ts_tablet_manager.cc:1397] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:59.609325 8391 raft_consensus.cc:348] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } }
I20260924 02:57:59.609473 8391 raft_consensus.cc:374] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:59.609515 8391 raft_consensus.cc:729] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71b2ecc8215f4e608fd14e6f07625b7a, State: Initialized, Role: FOLLOWER
I20260924 02:57:59.609624 8391 consensus_queue.cc:260] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } }
I20260924 02:57:59.609865 8391 ts_tablet_manager.cc:1428] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 02:57:59.609908 8086 heartbeater.cc:503] Master 127.0.98.62:46449 was elected leader, sending a full tablet report...
I20260924 02:57:59.610991 8395 raft_consensus.cc:348] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } }
I20260924 02:57:59.611102 8395 raft_consensus.cc:374] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:59.611121 8395 raft_consensus.cc:729] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47faee667646449abd37257851cb71a1, State: Initialized, Role: FOLLOWER
I20260924 02:57:59.611198 8395 consensus_queue.cc:260] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } }
I20260924 02:57:59.611413 8395 ts_tablet_manager.cc:1428] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 02:57:59.611485 8366 heartbeater.cc:503] Master 127.0.98.62:46449 was elected leader, sending a full tablet report...
I20260924 02:57:59.635520 8396 raft_consensus.cc:482] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:57:59.635622 8396 raft_consensus.cc:504] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } }
I20260924 02:57:59.635949 8396 leader_election.cc:288] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71b2ecc8215f4e608fd14e6f07625b7a (127.0.98.1:37615), 47faee667646449abd37257851cb71a1 (127.0.98.3:46189)
I20260924 02:57:59.638698 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.636177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48105 (local address 127.0.98.1:37615)
0924 02:57:59.636230 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.636232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.636239 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:57:59.636301 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.636302 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.636313 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.636359 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:59.636531 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.637010 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.638034 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.638378 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.638437 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.638448 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.638598 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.638635 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 02:57:59.638734 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.636391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52085 (local address 127.0.98.3:46189)
0924 02:57:59.636436 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.636438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.636445 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:57:59.636465 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.636466 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.636476 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.636517 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:59.636683 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.637125 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.638077 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.638419 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.638475 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.638484 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.638640 (+ 156us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.638671 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 02:57:59.638760 8119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.636124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37615 (local address 127.0.98.2:48105)
0924 02:57:59.636174 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:59.636185 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:59.636234 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:59.636388 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:59.636389 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:59.636391 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:59.636458 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.636459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.637082 (+ 623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.637083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.637955 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.637958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.638448 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.638449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.638542 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.638545 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:59.638587 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 02:57:59.638627 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 02:57:59.638763 8400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.636219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46189 (local address 127.0.98.2:52085)
0924 02:57:59.636366 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:59.636374 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:59.636409 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:59.636557 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:59.636557 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:59.636559 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:59.636631 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.636632 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.637190 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.637191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.638006 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.638008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.638482 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.638483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.638579 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.638583 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:57:59.638629 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 02:57:59.638663 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":106,"threads_started":1}
I20260924 02:57:59.639081 8040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae932bce29794bc991b31770af10ae3a" candidate_uuid: "b738799e612c4554a4ad08102c417de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" is_pre_election: true
I20260924 02:57:59.639169 8320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae932bce29794bc991b31770af10ae3a" candidate_uuid: "b738799e612c4554a4ad08102c417de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47faee667646449abd37257851cb71a1" is_pre_election: true
I20260924 02:57:59.639235 8040 raft_consensus.cc:2454] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b738799e612c4554a4ad08102c417de0 in term 0.
I20260924 02:57:59.639287 8320 raft_consensus.cc:2454] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b738799e612c4554a4ad08102c417de0 in term 0.
I20260924 02:57:59.639456 8113 leader_election.cc:302] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71b2ecc8215f4e608fd14e6f07625b7a, b738799e612c4554a4ad08102c417de0; no voters:
I20260924 02:57:59.639567 8396 raft_consensus.cc:2781] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:57:59.639644 8396 raft_consensus.cc:482] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:57:59.639673 8396 raft_consensus.cc:3037] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:59.640525 8396 raft_consensus.cc:504] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } }
I20260924 02:57:59.640635 8396 leader_election.cc:288] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [CANDIDATE]: Term 1 election: Requested vote from peers 71b2ecc8215f4e608fd14e6f07625b7a (127.0.98.1:37615), 47faee667646449abd37257851cb71a1 (127.0.98.3:46189)
I20260924 02:57:59.640799 8320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae932bce29794bc991b31770af10ae3a" candidate_uuid: "b738799e612c4554a4ad08102c417de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47faee667646449abd37257851cb71a1"
I20260924 02:57:59.640779 8040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ae932bce29794bc991b31770af10ae3a" candidate_uuid: "b738799e612c4554a4ad08102c417de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71b2ecc8215f4e608fd14e6f07625b7a"
I20260924 02:57:59.640868 8320 raft_consensus.cc:3037] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:59.640869 8040 raft_consensus.cc:3037] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:59.641707 8320 raft_consensus.cc:2454] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b738799e612c4554a4ad08102c417de0 in term 1.
I20260924 02:57:59.641707 8040 raft_consensus.cc:2454] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b738799e612c4554a4ad08102c417de0 in term 1.
I20260924 02:57:59.641849 8113 leader_election.cc:302] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71b2ecc8215f4e608fd14e6f07625b7a, b738799e612c4554a4ad08102c417de0; no voters:
I20260924 02:57:59.641934 8396 raft_consensus.cc:2781] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:59.642077 8396 raft_consensus.cc:686] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [term 1 LEADER]: Becoming Leader. State: Replica: b738799e612c4554a4ad08102c417de0, State: Running, Role: LEADER
I20260924 02:57:59.642168 8396 consensus_queue.cc:237] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } }
I20260924 02:57:59.642976 7883 catalog_manager.cc:5495] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b738799e612c4554a4ad08102c417de0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b738799e612c4554a4ad08102c417de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } health_report { overall_health: HEALTHY } } }
I20260924 02:57:59.657938 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.655894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51982 (local address 127.0.98.2:46053)
0924 02:57:59.655944 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.655946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.655953 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:57:59.656004 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.656005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.656022 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.656074 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:59.656177 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.656622 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.657310 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.657521 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.657566 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.657693 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:59.657784 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:59.657839 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.657853 (+ 14us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.657869 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 02:57:59.659634 8320 raft_consensus.cc:1261] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1 [term 1 FOLLOWER]: Refusing update from remote peer b738799e612c4554a4ad08102c417de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:59.659636 8040 raft_consensus.cc:1261] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Refusing update from remote peer b738799e612c4554a4ad08102c417de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:59.659924 8396 consensus_queue.cc:1035] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:59.660055 8401 consensus_queue.cc:1035] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:59.662489 8406 mvcc.cc:205] Tried to move back new op lower bound from 7332735711883538432 to 7332735711814868992. Current Snapshot: MvccSnapshot[applied={T|T < 7332735711883538432}]
I20260924 02:57:59.664579 7882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:59.664667 7882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:59.665315 7882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9: Sending DeleteTablet for 3 replicas of tablet ae932bce29794bc991b31770af10ae3a
I20260924 02:57:59.665582 8020 tablet_service.cc:1514] Processing DeleteTablet for tablet ae932bce29794bc991b31770af10ae3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55534
I20260924 02:57:59.665625 8160 tablet_service.cc:1514] Processing DeleteTablet for tablet ae932bce29794bc991b31770af10ae3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51970
I20260924 02:57:59.665768 8300 tablet_service.cc:1514] Processing DeleteTablet for tablet ae932bce29794bc991b31770af10ae3a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34628
I20260924 02:57:59.665892 8414 tablet_replica.cc:330] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0: stopping tablet replica
I20260924 02:57:59.665977 8415 tablet_replica.cc:330] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1: stopping tablet replica
I20260924 02:57:59.665995 8414 raft_consensus.cc:2229] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 02:57:59.666056 8415 raft_consensus.cc:2229] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:59.666075 8414 raft_consensus.cc:2258] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:59.666141 7882 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53476:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 02:57:59.666208 8413 tablet_replica.cc:330] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a: stopping tablet replica
W20260924 02:57:59.666281 7882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:57:59.666303 8413 raft_consensus.cc:2229] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:59.666404 8413 raft_consensus.cc:2258] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:59.666585 8414 ts_tablet_manager.cc:1905] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:59.666818 8413 ts_tablet_manager.cc:1905] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:59.667446 8160 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3dda8c3fb248ef9f98ed3102778d6c (DEFAULT_TABLE table=test-table [id=7dc5927629c749fea991988c19613403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:59.667552 8160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3dda8c3fb248ef9f98ed3102778d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:59.668336 8414 ts_tablet_manager.cc:1918] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:57:59.668465 8414 log.cc:1196] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/ae932bce29794bc991b31770af10ae3a
I20260924 02:57:59.668552 8413 ts_tablet_manager.cc:1918] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:57:59.668593 8020 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3dda8c3fb248ef9f98ed3102778d6c (DEFAULT_TABLE table=test-table [id=7dc5927629c749fea991988c19613403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:59.668627 8413 log.cc:1196] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/wal/wals/ae932bce29794bc991b31770af10ae3a
I20260924 02:57:59.668686 8020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3dda8c3fb248ef9f98ed3102778d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:59.668857 8413 ts_tablet_manager.cc:1939] T ae932bce29794bc991b31770af10ae3a P 71b2ecc8215f4e608fd14e6f07625b7a: Deleting consensus metadata
I20260924 02:57:59.668977 8414 ts_tablet_manager.cc:1939] T ae932bce29794bc991b31770af10ae3a P b738799e612c4554a4ad08102c417de0: Deleting consensus metadata
I20260924 02:57:59.669247 7869 catalog_manager.cc:4841] TS 71b2ecc8215f4e608fd14e6f07625b7a (127.0.98.1:37615): tablet ae932bce29794bc991b31770af10ae3a (table test-table [id=27b2bd130ffb49e391c09f9e8fe5508d]) successfully deleted
I20260924 02:57:59.669706 8391 tablet_bootstrap.cc:492] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a: Bootstrap starting.
I20260924 02:57:59.669750 8300 tablet_service.cc:1467] Processing CreateTablet for tablet 7d3dda8c3fb248ef9f98ed3102778d6c (DEFAULT_TABLE table=test-table [id=7dc5927629c749fea991988c19613403]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:59.669852 8300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7d3dda8c3fb248ef9f98ed3102778d6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:59.670111 8391 tablet_bootstrap.cc:654] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:59.670697 8391 tablet_bootstrap.cc:492] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a: No bootstrap required, opened a new log
I20260924 02:57:59.670753 8391 ts_tablet_manager.cc:1397] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:59.670893 8391 raft_consensus.cc:348] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.670953 8395 tablet_bootstrap.cc:492] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1: Bootstrap starting.
I20260924 02:57:59.670967 8391 raft_consensus.cc:374] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:59.670987 8391 raft_consensus.cc:729] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71b2ecc8215f4e608fd14e6f07625b7a, State: Initialized, Role: FOLLOWER
I20260924 02:57:59.671023 8391 consensus_queue.cc:260] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.671099 8391 ts_tablet_manager.cc:1428] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:59.671316 8395 tablet_bootstrap.cc:654] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:59.671597 8392 tablet_bootstrap.cc:492] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0: Bootstrap starting.
I20260924 02:57:59.671932 8392 tablet_bootstrap.cc:654] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:59.672328 7870 catalog_manager.cc:4841] TS b738799e612c4554a4ad08102c417de0 (127.0.98.2:46053): tablet ae932bce29794bc991b31770af10ae3a (table test-table [id=27b2bd130ffb49e391c09f9e8fe5508d]) successfully deleted
I20260924 02:57:59.672559 8392 tablet_bootstrap.cc:492] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0: No bootstrap required, opened a new log
I20260924 02:57:59.672596 8392 ts_tablet_manager.cc:1397] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:59.672693 8392 raft_consensus.cc:348] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.672742 8392 raft_consensus.cc:374] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:59.672755 8392 raft_consensus.cc:729] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b738799e612c4554a4ad08102c417de0, State: Initialized, Role: FOLLOWER
I20260924 02:57:59.672783 8392 consensus_queue.cc:260] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.672860 8392 ts_tablet_manager.cc:1428] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:59.673031 8395 tablet_bootstrap.cc:492] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1: No bootstrap required, opened a new log
I20260924 02:57:59.673067 8395 ts_tablet_manager.cc:1397] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:57:59.673434 8395 raft_consensus.cc:348] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.673504 8395 raft_consensus.cc:374] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:59.673520 8395 raft_consensus.cc:729] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47faee667646449abd37257851cb71a1, State: Initialized, Role: FOLLOWER
I20260924 02:57:59.673553 8395 consensus_queue.cc:260] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.673635 8395 ts_tablet_manager.cc:1428] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 02:57:59.673802 8415 raft_consensus.cc:2258] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:59.674541 8415 ts_tablet_manager.cc:1905] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:59.675825 8415 ts_tablet_manager.cc:1918] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:57:59.675876 8415 log.cc:1196] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/ae932bce29794bc991b31770af10ae3a
I20260924 02:57:59.676038 8415 ts_tablet_manager.cc:1939] T ae932bce29794bc991b31770af10ae3a P 47faee667646449abd37257851cb71a1: Deleting consensus metadata
I20260924 02:57:59.676337 7869 catalog_manager.cc:4841] TS 47faee667646449abd37257851cb71a1 (127.0.98.3:46189): tablet ae932bce29794bc991b31770af10ae3a (table test-table [id=27b2bd130ffb49e391c09f9e8fe5508d]) successfully deleted
I20260924 02:57:59.741211 8396 raft_consensus.cc:482] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:57:59.741319 8396 raft_consensus.cc:504] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.741473 8396 leader_election.cc:288] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71b2ecc8215f4e608fd14e6f07625b7a (127.0.98.1:37615), 47faee667646449abd37257851cb71a1 (127.0.98.3:46189)
I20260924 02:57:59.741680 8040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3dda8c3fb248ef9f98ed3102778d6c" candidate_uuid: "b738799e612c4554a4ad08102c417de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" is_pre_election: true
I20260924 02:57:59.741700 8320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3dda8c3fb248ef9f98ed3102778d6c" candidate_uuid: "b738799e612c4554a4ad08102c417de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47faee667646449abd37257851cb71a1" is_pre_election: true
I20260924 02:57:59.741776 8040 raft_consensus.cc:2454] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b738799e612c4554a4ad08102c417de0 in term 0.
I20260924 02:57:59.741796 8320 raft_consensus.cc:2454] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b738799e612c4554a4ad08102c417de0 in term 0.
I20260924 02:57:59.741927 8113 leader_election.cc:302] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71b2ecc8215f4e608fd14e6f07625b7a, b738799e612c4554a4ad08102c417de0; no voters:
I20260924 02:57:59.742033 8396 raft_consensus.cc:2781] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:57:59.742089 8396 raft_consensus.cc:482] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:57:59.742115 8396 raft_consensus.cc:3037] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:59.743031 8396 raft_consensus.cc:504] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.743160 8396 leader_election.cc:288] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [CANDIDATE]: Term 1 election: Requested vote from peers 71b2ecc8215f4e608fd14e6f07625b7a (127.0.98.1:37615), 47faee667646449abd37257851cb71a1 (127.0.98.3:46189)
I20260924 02:57:59.743314 8040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3dda8c3fb248ef9f98ed3102778d6c" candidate_uuid: "b738799e612c4554a4ad08102c417de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71b2ecc8215f4e608fd14e6f07625b7a"
I20260924 02:57:59.743330 8320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7d3dda8c3fb248ef9f98ed3102778d6c" candidate_uuid: "b738799e612c4554a4ad08102c417de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47faee667646449abd37257851cb71a1"
I20260924 02:57:59.743397 8040 raft_consensus.cc:3037] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:59.743399 8320 raft_consensus.cc:3037] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:59.744154 8040 raft_consensus.cc:2454] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b738799e612c4554a4ad08102c417de0 in term 1.
I20260924 02:57:59.744189 8320 raft_consensus.cc:2454] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b738799e612c4554a4ad08102c417de0 in term 1.
I20260924 02:57:59.744313 8113 leader_election.cc:302] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71b2ecc8215f4e608fd14e6f07625b7a, b738799e612c4554a4ad08102c417de0; no voters:
I20260924 02:57:59.744418 8396 raft_consensus.cc:2781] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:59.744467 8396 raft_consensus.cc:686] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [term 1 LEADER]: Becoming Leader. State: Replica: b738799e612c4554a4ad08102c417de0, State: Running, Role: LEADER
I20260924 02:57:59.744515 8396 consensus_queue.cc:237] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.745066 7883 catalog_manager.cc:5495] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b738799e612c4554a4ad08102c417de0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b738799e612c4554a4ad08102c417de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } health_report { overall_health: UNKNOWN } } }
W20260924 02:57:59.767802 8087 tablet.cc:2369] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 02:57:59 dist-test-slave-sb6p krb5kdc[7838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218679, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.62@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 02:57:59.785631 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.779267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53484 (local address 127.0.98.62:46449)
0924 02:57:59.779327 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.779329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.779484 (+ 155us) server_negotiation.cc:407] Connection header received
0924 02:57:59.779514 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.779515 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.779524 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.779587 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:59.780101 (+ 514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.780601 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.781347 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.781528 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.781583 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.783497 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:59.783498 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:59.783498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:59.783504 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:59.784703 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.785053 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.785054 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.785054 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.785101 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.785274 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.785274 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.785274 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.785361 (+ 87us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:59.785405 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.785507 (+ 102us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.785565 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 02:57:59.785735 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.779179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46449 (local address 127.0.0.1:53484)
0924 02:57:59.779392 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:59.779403 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:59.779461 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:59.779599 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:59.779600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:59.779952 (+ 352us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:57:59.779999 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.780002 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.780666 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.780667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.781224 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.781226 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.781591 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.781592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.781625 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.782089 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:57:59.782094 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:57:59.783425 (+ 1331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:57:59.784800 (+ 1375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:59.784806 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:59.784808 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:59.784991 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:59.785159 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:57:59.785160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:57:59.785160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:57:59.785227 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:57:59.785401 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:57:59.785406 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:57:59.785446 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 02:57:59.785492 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 02:57:59.785614 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":40,"thread_start_us":144,"threads_started":1}
I20260924 02:57:59.787532 7883 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:53484:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260924 02:57:59.787716 7883 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:57:59.789119 8020 tablet_service.cc:1467] Processing CreateTablet for tablet 66d0c9e8acb34d159d7d23a16420df62 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5be77f55ff024c678fce03e6088b3c7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 02:57:59.789155 8300 tablet_service.cc:1467] Processing CreateTablet for tablet 66d0c9e8acb34d159d7d23a16420df62 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5be77f55ff024c678fce03e6088b3c7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 02:57:59.789271 8300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66d0c9e8acb34d159d7d23a16420df62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:59.789327 8020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66d0c9e8acb34d159d7d23a16420df62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:59.790726 8395 tablet_bootstrap.cc:492] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1: Bootstrap starting.
I20260924 02:57:59.791152 8395 tablet_bootstrap.cc:654] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:59.791498 8391 tablet_bootstrap.cc:492] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a: Bootstrap starting.
I20260924 02:57:59.791765 8395 tablet_bootstrap.cc:492] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1: No bootstrap required, opened a new log
I20260924 02:57:59.791808 8395 ts_tablet_manager.cc:1397] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:59.791869 8391 tablet_bootstrap.cc:654] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:59.791954 8395 raft_consensus.cc:348] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.792021 8395 raft_consensus.cc:374] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:59.792039 8395 raft_consensus.cc:729] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47faee667646449abd37257851cb71a1, State: Initialized, Role: FOLLOWER
I20260924 02:57:59.792080 8395 consensus_queue.cc:260] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.792161 8395 ts_tablet_manager.cc:1428] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:59.792243 8399 tablet_replica.cc:439] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } }
I20260924 02:57:59.792377 8399 tablet_replica.cc:442] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:59.792923 8160 tablet_service.cc:1467] Processing CreateTablet for tablet 66d0c9e8acb34d159d7d23a16420df62 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=5be77f55ff024c678fce03e6088b3c7b]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260924 02:57:59.793064 8160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 66d0c9e8acb34d159d7d23a16420df62. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:59.794406 8392 tablet_bootstrap.cc:492] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0: Bootstrap starting.
I20260924 02:57:59.794772 8392 tablet_bootstrap.cc:654] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:59.794831 8391 tablet_bootstrap.cc:492] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a: No bootstrap required, opened a new log
I20260924 02:57:59.794878 8391 ts_tablet_manager.cc:1397] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 02:57:59.795019 8391 raft_consensus.cc:348] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.795087 8391 raft_consensus.cc:374] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:59.795106 8391 raft_consensus.cc:729] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71b2ecc8215f4e608fd14e6f07625b7a, State: Initialized, Role: FOLLOWER
I20260924 02:57:59.795153 8391 consensus_queue.cc:260] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.795240 8391 ts_tablet_manager.cc:1428] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:59.795485 8392 tablet_bootstrap.cc:492] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0: No bootstrap required, opened a new log
I20260924 02:57:59.795532 8392 ts_tablet_manager.cc:1397] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:59.795503 8398 tablet_replica.cc:439] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } }
I20260924 02:57:59.795580 8398 tablet_replica.cc:442] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:59.795630 8392 raft_consensus.cc:348] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.795693 8392 raft_consensus.cc:374] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:59.795715 8392 raft_consensus.cc:729] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b738799e612c4554a4ad08102c417de0, State: Initialized, Role: FOLLOWER
I20260924 02:57:59.795751 8392 consensus_queue.cc:260] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.795832 8392 ts_tablet_manager.cc:1428] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:59.795866 8396 tablet_replica.cc:439] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } }
I20260924 02:57:59.795956 8396 tablet_replica.cc:442] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0: This TxnStatusTablet replica's current role is: FOLLOWER
W20260924 02:57:59.805588 8367 tablet.cc:2369] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 02:57:59.873545 8399 raft_consensus.cc:482] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:57:59.873633 8399 raft_consensus.cc:504] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.873925 8399 leader_election.cc:288] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71b2ecc8215f4e608fd14e6f07625b7a (127.0.98.1:37615), b738799e612c4554a4ad08102c417de0 (127.0.98.2:46053)
I20260924 02:57:59.876731 8429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.874104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46053 (local address 127.0.98.3:52299)
0924 02:57:59.874243 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:59.874251 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:59.874305 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:59.874448 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:59.874448 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:59.874450 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:59.874548 (+ 98us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.874550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.875122 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.875123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.875876 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.875878 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.876484 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.876485 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.876554 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.876558 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:57:59.876609 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 02:57:59.876644 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":71,"threads_started":1}
I20260924 02:57:59.876724 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.874218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52299 (local address 127.0.98.2:46053)
0924 02:57:59.874267 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.874268 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.874293 (+ 25us) server_negotiation.cc:407] Connection header received
0924 02:57:59.874361 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.874362 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.874371 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.874440 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:59.874603 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.875047 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.875975 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.876414 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.876465 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.876473 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.876622 (+ 149us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.876652 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 02:57:59.876890 8259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.874091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37615 (local address 127.0.98.3:36145)
0924 02:57:59.874127 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:59.874138 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:59.874183 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:59.874323 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:59.874323 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:59.874325 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:57:59.874398 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.874400 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.875074 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.875075 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.876100 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.876102 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.876677 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.876678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.876746 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.876749 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:57:59.876796 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 02:57:59.876834 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 02:57:59.876968 8388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.874140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36145 (local address 127.0.98.1:37615)
0924 02:57:59.874189 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.874191 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.874222 (+ 31us) server_negotiation.cc:407] Connection header received
0924 02:57:59.874251 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.874251 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.874260 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.874309 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:57:59.874460 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.875001 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.876182 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.876604 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.876661 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.876670 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.876867 (+ 197us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.876910 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 02:57:59.877144 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d0c9e8acb34d159d7d23a16420df62" candidate_uuid: "47faee667646449abd37257851cb71a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b738799e612c4554a4ad08102c417de0" is_pre_election: true
I20260924 02:57:59.877185 8040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d0c9e8acb34d159d7d23a16420df62" candidate_uuid: "47faee667646449abd37257851cb71a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" is_pre_election: true
I20260924 02:57:59.877256 8040 raft_consensus.cc:2454] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47faee667646449abd37257851cb71a1 in term 0.
I20260924 02:57:59.877259 8180 raft_consensus.cc:2454] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47faee667646449abd37257851cb71a1 in term 0.
I20260924 02:57:59.877421 8253 leader_election.cc:302] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47faee667646449abd37257851cb71a1, 71b2ecc8215f4e608fd14e6f07625b7a; no voters:
I20260924 02:57:59.877533 8399 raft_consensus.cc:2781] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:57:59.877568 8399 raft_consensus.cc:482] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:57:59.877586 8399 raft_consensus.cc:3037] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:59.878201 8399 raft_consensus.cc:504] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.878314 8399 leader_election.cc:288] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [CANDIDATE]: Term 1 election: Requested vote from peers 71b2ecc8215f4e608fd14e6f07625b7a (127.0.98.1:37615), b738799e612c4554a4ad08102c417de0 (127.0.98.2:46053)
I20260924 02:57:59.878450 8040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d0c9e8acb34d159d7d23a16420df62" candidate_uuid: "47faee667646449abd37257851cb71a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71b2ecc8215f4e608fd14e6f07625b7a"
I20260924 02:57:59.878538 8040 raft_consensus.cc:3037] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:59.878506 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "66d0c9e8acb34d159d7d23a16420df62" candidate_uuid: "47faee667646449abd37257851cb71a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b738799e612c4554a4ad08102c417de0"
I20260924 02:57:59.878602 8180 raft_consensus.cc:3037] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:57:59.879078 8040 raft_consensus.cc:2454] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47faee667646449abd37257851cb71a1 in term 1.
I20260924 02:57:59.879216 8180 raft_consensus.cc:2454] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47faee667646449abd37257851cb71a1 in term 1.
I20260924 02:57:59.879232 8253 leader_election.cc:302] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47faee667646449abd37257851cb71a1, 71b2ecc8215f4e608fd14e6f07625b7a; no voters:
I20260924 02:57:59.879369 8399 raft_consensus.cc:2781] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:57:59.879498 8399 raft_consensus.cc:686] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [term 1 LEADER]: Becoming Leader. State: Replica: 47faee667646449abd37257851cb71a1, State: Running, Role: LEADER
I20260924 02:57:59.879565 8399 consensus_queue.cc:237] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.879879 8430 tablet_replica.cc:439] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1: TxnStatusTablet state changed. Reason: New leader 47faee667646449abd37257851cb71a1. Latest consensus state: current_term: 1 leader_uuid: "47faee667646449abd37257851cb71a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } }
I20260924 02:57:59.880002 8430 tablet_replica.cc:442] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1: This TxnStatusTablet replica's current role is: LEADER
I20260924 02:57:59.880224 7883 catalog_manager.cc:5495] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47faee667646449abd37257851cb71a1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "47faee667646449abd37257851cb71a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } health_report { overall_health: HEALTHY } } }
I20260924 02:57:59.880496 8432 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260924 02:57:59.880595 8432 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260924 02:57:59.891418 8425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.889143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46189 (local address 127.0.0.1:34630)
0924 02:57:59.889205 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:59.889215 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:59.889258 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:59.889405 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:59.889406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:59.889407 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 02:57:59.889460 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.889463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.890037 (+ 574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.890038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.890631 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.890633 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.890954 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.890954 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.891017 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.891029 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 02:57:59.891255 (+ 226us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 02:57:59.891262 (+ 7us) client_negotiation.cc:769] Sending connection context
0924 02:57:59.891304 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 02:57:59.891353 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 02:57:59.891396 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.889211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34630 (local address 127.0.98.3:46189)
0924 02:57:59.889264 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.889266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.889274 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:57:59.889332 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.889333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.889351 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.889419 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:59.889514 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.889971 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.890699 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.890898 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.890950 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.891103 (+ 153us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:59.891202 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:59.891262 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.891306 (+ 44us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.891327 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 02:57:59.892475 8040 raft_consensus.cc:1261] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Refusing update from remote peer 47faee667646449abd37257851cb71a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:59.892491 8180 raft_consensus.cc:1261] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0 [term 1 FOLLOWER]: Refusing update from remote peer 47faee667646449abd37257851cb71a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:59.892727 8430 consensus_queue.cc:1035] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:59.892859 8399 consensus_queue.cc:1035] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:59.893349 8398 tablet_replica.cc:439] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a: TxnStatusTablet state changed. Reason: New leader 47faee667646449abd37257851cb71a1. Latest consensus state: current_term: 1 leader_uuid: "47faee667646449abd37257851cb71a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } }
I20260924 02:57:59.893436 8398 tablet_replica.cc:442] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:59.893424 8396 tablet_replica.cc:439] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0: TxnStatusTablet state changed. Reason: New leader 47faee667646449abd37257851cb71a1. Latest consensus state: current_term: 1 leader_uuid: "47faee667646449abd37257851cb71a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } }
I20260924 02:57:59.893501 8396 tablet_replica.cc:442] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:59.894096 8430 tablet_replica.cc:439] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47faee667646449abd37257851cb71a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } }
I20260924 02:57:59.894164 8430 tablet_replica.cc:442] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1: This TxnStatusTablet replica's current role is: LEADER
I20260924 02:57:59.894565 8404 mvcc.cc:205] Tried to move back new op lower bound from 7332735712837537792 to 7332735712787152896. Current Snapshot: MvccSnapshot[applied={T|T < 7332735712837537792 or (T in {7332735712837537792})}]
I20260924 02:57:59.894671 8396 tablet_replica.cc:439] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47faee667646449abd37257851cb71a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } }
I20260924 02:57:59.894735 8396 tablet_replica.cc:442] T 66d0c9e8acb34d159d7d23a16420df62 P b738799e612c4554a4ad08102c417de0: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:59.895506 8430 tablet_replica.cc:439] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47faee667646449abd37257851cb71a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } }
I20260924 02:57:59.895592 8430 tablet_replica.cc:442] T 66d0c9e8acb34d159d7d23a16420df62 P 47faee667646449abd37257851cb71a1: This TxnStatusTablet replica's current role is: LEADER
I20260924 02:57:59.895681 8398 tablet_replica.cc:439] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47faee667646449abd37257851cb71a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } } }
I20260924 02:57:59.895702 8405 mvcc.cc:205] Tried to move back new op lower bound from 7332735712837537792 to 7332735712787152896. Current Snapshot: MvccSnapshot[applied={T|T < 7332735712837537792}]
I20260924 02:57:59.895750 8398 tablet_replica.cc:442] T 66d0c9e8acb34d159d7d23a16420df62 P 71b2ecc8215f4e608fd14e6f07625b7a: This TxnStatusTablet replica's current role is: FOLLOWER
I20260924 02:57:59.899638 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.897578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57827 (local address 127.0.98.3:46189)
0924 02:57:59.897615 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.897617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.897627 (+ 10us) server_negotiation.cc:407] Connection header received
0924 02:57:59.897685 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.897685 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.897693 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.897741 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:59.897826 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.898265 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.899069 (+ 804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.899256 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.899306 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.899442 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:59.899472 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:59.899511 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.899560 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.899576 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 02:57:59.899660 8233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.897522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:46189 (local address 127.0.98.2:57827)
0924 02:57:59.897591 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:59.897603 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:59.897642 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:59.897744 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:59.897745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:59.897746 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 02:57:59.897785 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.897787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.898331 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.898333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.899001 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.899003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.899310 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.899311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.899364 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.899378 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 02:57:59.899517 (+ 139us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 02:57:59.899519 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 02:57:59.899557 (+ 38us) client_negotiation.cc:240] Negotiation successful
0924 02:57:59.899601 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 02:57:59.901364 8040 raft_consensus.cc:1261] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Refusing update from remote peer b738799e612c4554a4ad08102c417de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:59.901364 8320 raft_consensus.cc:1261] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1 [term 1 FOLLOWER]: Refusing update from remote peer b738799e612c4554a4ad08102c417de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:57:59.901537 8401 consensus_queue.cc:1035] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:59.901654 8396 consensus_queue.cc:1035] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:57:59.909627 8372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.907078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46053 (local address 127.0.98.3:57887)
0924 02:57:59.907142 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0924 02:57:59.907153 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:57:59.907193 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:57:59.907329 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:57:59.907330 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:57:59.907331 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0924 02:57:59.907386 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.907387 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.908263 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.908264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.908880 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:57:59.908882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.909308 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.909309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:57:59.909361 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.909374 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0924 02:57:59.909520 (+ 146us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0924 02:57:59.909521 (+ 1us) client_negotiation.cc:769] Sending connection context
0924 02:57:59.909549 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 02:57:59.909578 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 02:57:59.909642 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.907210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:57887 (local address 127.0.98.2:46053)
0924 02:57:59.907242 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.907244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.907250 (+ 6us) server_negotiation.cc:407] Connection header received
0924 02:57:59.907268 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.907269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.907278 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.907324 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:57:59.907460 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.908196 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.908956 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.909250 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.909311 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.909440 (+ 129us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:57:59.909472 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:57:59.909518 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.909554 (+ 36us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.909573 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 02:57:59.915261 7883 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:59.915359 7883 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53476:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:57:59.915882 7883 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9: Sending DeleteTablet for 3 replicas of tablet 7d3dda8c3fb248ef9f98ed3102778d6c
I20260924 02:57:59.916132 8160 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3dda8c3fb248ef9f98ed3102778d6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51970
I20260924 02:57:59.916177 8300 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3dda8c3fb248ef9f98ed3102778d6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34628
I20260924 02:57:59.916177 8020 tablet_service.cc:1514] Processing DeleteTablet for tablet 7d3dda8c3fb248ef9f98ed3102778d6c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:57:59 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55534
I20260924 02:57:59.916220 8414 tablet_replica.cc:330] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0: stopping tablet replica
I20260924 02:57:59.916258 8413 tablet_replica.cc:330] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a: stopping tablet replica
I20260924 02:57:59.916266 8414 raft_consensus.cc:2229] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 02:57:59.916261 8415 tablet_replica.cc:330] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1: stopping tablet replica
I20260924 02:57:59.916296 8413 raft_consensus.cc:2229] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:59.916308 8415 raft_consensus.cc:2229] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:57:59.916322 8413 raft_consensus.cc:2258] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:59.916316 8414 raft_consensus.cc:2258] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:59.916337 8415 raft_consensus.cc:2258] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:57:59.916596 8415 ts_tablet_manager.cc:1905] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:59.916827 8414 ts_tablet_manager.cc:1905] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:59.917938 8413 ts_tablet_manager.cc:1905] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:57:59.918058 8414 ts_tablet_manager.cc:1918] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 02:57:59.918128 8414 log.cc:1196] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/7d3dda8c3fb248ef9f98ed3102778d6c
I20260924 02:57:59.918121 8415 ts_tablet_manager.cc:1918] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 02:57:59.918169 8415 log.cc:1196] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/7d3dda8c3fb248ef9f98ed3102778d6c
I20260924 02:57:59.918332 8414 ts_tablet_manager.cc:1939] T 7d3dda8c3fb248ef9f98ed3102778d6c P b738799e612c4554a4ad08102c417de0: Deleting consensus metadata
I20260924 02:57:59.918346 8415 ts_tablet_manager.cc:1939] T 7d3dda8c3fb248ef9f98ed3102778d6c P 47faee667646449abd37257851cb71a1: Deleting consensus metadata
I20260924 02:57:59.919274 8413 ts_tablet_manager.cc:1918] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 02:57:59.919329 8413 log.cc:1196] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/wal/wals/7d3dda8c3fb248ef9f98ed3102778d6c
I20260924 02:57:59.919494 8413 ts_tablet_manager.cc:1939] T 7d3dda8c3fb248ef9f98ed3102778d6c P 71b2ecc8215f4e608fd14e6f07625b7a: Deleting consensus metadata
I20260924 02:57:59.919661 7869 catalog_manager.cc:4841] TS 47faee667646449abd37257851cb71a1 (127.0.98.3:46189): tablet 7d3dda8c3fb248ef9f98ed3102778d6c (table test-table [id=7dc5927629c749fea991988c19613403]) successfully deleted
I20260924 02:57:59.919726 7870 catalog_manager.cc:4841] TS b738799e612c4554a4ad08102c417de0 (127.0.98.2:46053): tablet 7d3dda8c3fb248ef9f98ed3102778d6c (table test-table [id=7dc5927629c749fea991988c19613403]) successfully deleted
I20260924 02:57:59.919818 7869 catalog_manager.cc:4841] TS 71b2ecc8215f4e608fd14e6f07625b7a (127.0.98.1:37615): tablet 7d3dda8c3fb248ef9f98ed3102778d6c (table test-table [id=7dc5927629c749fea991988c19613403]) successfully deleted
Sep 24 02:57:59 dist-test-slave-sb6p krb5kdc[7838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 24 02:57:59 dist-test-slave-sb6p krb5kdc[7838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:57:59 dist-test-slave-sb6p krb5kdc[7838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 02:57:59.940251 7951 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:57:59.934471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53486 (local address 127.0.98.62:46449)
0924 02:57:59.934554 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.934556 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.934647 (+ 91us) server_negotiation.cc:407] Connection header received
0924 02:57:59.934721 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.934722 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.934731 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.934778 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:59.935459 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.935951 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.937045 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.937264 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.937310 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.940188 (+ 2878us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53486: BlockingRecv error: recv got EOF from 127.0.0.1:53486 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 02:57:59 dist-test-slave-sb6p krb5kdc[7838](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM
I20260924 02:57:59.947818 7951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:57:59.941066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53500 (local address 127.0.98.62:46449)
0924 02:57:59.941107 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 02:57:59.941109 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:57:59.941169 (+ 60us) server_negotiation.cc:407] Connection header received
0924 02:57:59.941215 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:57:59.941216 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:57:59.941226 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:57:59.941288 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:57:59.941670 (+ 382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.942403 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.943087 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:57:59.943343 (+ 256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:57:59.943412 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:57:59.945298 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:57:59.945300 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:57:59.945300 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:57:59.945307 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:57:59.946727 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.947016 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.947017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.947017 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.947076 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:57:59.947226 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:57:59.947227 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:57:59.947227 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:57:59.947348 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:57:59.947467 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0924 02:57:59.947681 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 02:57:59.947747 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 02:57:59.949167 7883 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 02:57:59.949302 7883 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:57:59.950507 8300 tablet_service.cc:1467] Processing CreateTablet for tablet 35405a51afe94882ad979d12e6f8e795 (DEFAULT_TABLE table=test-table [id=58455af841d444f1a848fa739cba3b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:59.950543 8160 tablet_service.cc:1467] Processing CreateTablet for tablet 35405a51afe94882ad979d12e6f8e795 (DEFAULT_TABLE table=test-table [id=58455af841d444f1a848fa739cba3b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:59.950656 8300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35405a51afe94882ad979d12e6f8e795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:59.950675 8160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35405a51afe94882ad979d12e6f8e795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:59.951563 8395 tablet_bootstrap.cc:492] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1: Bootstrap starting.
I20260924 02:57:59.951622 8020 tablet_service.cc:1467] Processing CreateTablet for tablet 35405a51afe94882ad979d12e6f8e795 (DEFAULT_TABLE table=test-table [id=58455af841d444f1a848fa739cba3b80]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:57:59.951722 8020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35405a51afe94882ad979d12e6f8e795. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:57:59.951860 8395 tablet_bootstrap.cc:654] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:59.952819 8392 tablet_bootstrap.cc:492] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0: Bootstrap starting.
I20260924 02:57:59.952817 8391 tablet_bootstrap.cc:492] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a: Bootstrap starting.
I20260924 02:57:59.953147 8392 tablet_bootstrap.cc:654] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:59.953230 8391 tablet_bootstrap.cc:654] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a: Neither blocks nor log segments found. Creating new log.
I20260924 02:57:59.953619 8392 tablet_bootstrap.cc:492] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0: No bootstrap required, opened a new log
I20260924 02:57:59.953672 8392 ts_tablet_manager.cc:1397] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:59.953771 8392 raft_consensus.cc:348] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.953827 8392 raft_consensus.cc:374] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:59.953845 8392 raft_consensus.cc:729] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b738799e612c4554a4ad08102c417de0, State: Initialized, Role: FOLLOWER
I20260924 02:57:59.953851 8391 tablet_bootstrap.cc:492] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a: No bootstrap required, opened a new log
I20260924 02:57:59.953881 8392 consensus_queue.cc:260] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.953909 8391 ts_tablet_manager.cc:1397] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 02:57:59.953971 8392 ts_tablet_manager.cc:1428] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:57:59.954071 8391 raft_consensus.cc:348] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.954138 8391 raft_consensus.cc:374] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:59.954161 8391 raft_consensus.cc:729] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71b2ecc8215f4e608fd14e6f07625b7a, State: Initialized, Role: FOLLOWER
I20260924 02:57:59.954195 8391 consensus_queue.cc:260] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.954284 8391 ts_tablet_manager.cc:1428] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 02:57:59.954345 8395 tablet_bootstrap.cc:492] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1: No bootstrap required, opened a new log
I20260924 02:57:59.954388 8395 ts_tablet_manager.cc:1397] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.000s
I20260924 02:57:59.954571 8395 raft_consensus.cc:348] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.954638 8395 raft_consensus.cc:374] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:57:59.954658 8395 raft_consensus.cc:729] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47faee667646449abd37257851cb71a1, State: Initialized, Role: FOLLOWER
I20260924 02:57:59.954695 8395 consensus_queue.cc:260] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:57:59.954773 8395 ts_tablet_manager.cc:1428] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 02:58:00.133296 8437 raft_consensus.cc:482] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:58:00.133392 8437 raft_consensus.cc:504] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:58:00.133556 8437 leader_election.cc:288] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71b2ecc8215f4e608fd14e6f07625b7a (127.0.98.1:37615), b738799e612c4554a4ad08102c417de0 (127.0.98.2:46053)
I20260924 02:58:00.133759 8040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35405a51afe94882ad979d12e6f8e795" candidate_uuid: "47faee667646449abd37257851cb71a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" is_pre_election: true
I20260924 02:58:00.133759 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35405a51afe94882ad979d12e6f8e795" candidate_uuid: "47faee667646449abd37257851cb71a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b738799e612c4554a4ad08102c417de0" is_pre_election: true
I20260924 02:58:00.133857 8040 raft_consensus.cc:2454] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47faee667646449abd37257851cb71a1 in term 0.
I20260924 02:58:00.133857 8180 raft_consensus.cc:2454] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47faee667646449abd37257851cb71a1 in term 0.
I20260924 02:58:00.134034 8254 leader_election.cc:302] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47faee667646449abd37257851cb71a1, b738799e612c4554a4ad08102c417de0; no voters:
I20260924 02:58:00.134121 8437 raft_consensus.cc:2781] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:58:00.134153 8437 raft_consensus.cc:482] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:58:00.134171 8437 raft_consensus.cc:3037] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:58:00.135079 8437 raft_consensus.cc:504] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:58:00.135202 8437 leader_election.cc:288] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [CANDIDATE]: Term 1 election: Requested vote from peers 71b2ecc8215f4e608fd14e6f07625b7a (127.0.98.1:37615), b738799e612c4554a4ad08102c417de0 (127.0.98.2:46053)
I20260924 02:58:00.135347 8180 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35405a51afe94882ad979d12e6f8e795" candidate_uuid: "47faee667646449abd37257851cb71a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b738799e612c4554a4ad08102c417de0"
I20260924 02:58:00.135356 8040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35405a51afe94882ad979d12e6f8e795" candidate_uuid: "47faee667646449abd37257851cb71a1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71b2ecc8215f4e608fd14e6f07625b7a"
I20260924 02:58:00.135427 8180 raft_consensus.cc:3037] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:58:00.135442 8040 raft_consensus.cc:3037] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:58:00.135993 8180 raft_consensus.cc:2454] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47faee667646449abd37257851cb71a1 in term 1.
I20260924 02:58:00.136018 8040 raft_consensus.cc:2454] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47faee667646449abd37257851cb71a1 in term 1.
I20260924 02:58:00.136159 8253 leader_election.cc:302] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47faee667646449abd37257851cb71a1, 71b2ecc8215f4e608fd14e6f07625b7a; no voters:
I20260924 02:58:00.136236 8437 raft_consensus.cc:2781] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:58:00.136293 8437 raft_consensus.cc:686] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [term 1 LEADER]: Becoming Leader. State: Replica: 47faee667646449abd37257851cb71a1, State: Running, Role: LEADER
I20260924 02:58:00.136348 8437 consensus_queue.cc:237] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:58:00.136904 7883 catalog_manager.cc:5495] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47faee667646449abd37257851cb71a1 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "47faee667646449abd37257851cb71a1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } health_report { overall_health: HEALTHY } } }
W20260924 02:58:00.166716 8227 tablet.cc:2369] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 02:58:00.180369 8390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:00.178253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34644 (local address 127.0.98.3:46189)
0924 02:58:00.178323 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:00.178325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:00.178334 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:58:00.178378 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:00.178379 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:00.178389 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:00.178455 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:58:00.178597 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:00.179062 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:00.179781 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:00.179976 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:00.180030 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:00.180150 (+ 120us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:58:00.180183 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:58:00.180223 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:00.180272 (+ 49us) server_negotiation.cc:299] Negotiation successful
0924 02:58:00.180291 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 02:58:00.181193 8040 raft_consensus.cc:1261] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Refusing update from remote peer 47faee667646449abd37257851cb71a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:58:00.181186 8180 raft_consensus.cc:1261] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0 [term 1 FOLLOWER]: Refusing update from remote peer 47faee667646449abd37257851cb71a1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:58:00.181413 8437 consensus_queue.cc:1035] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:58:00.181536 8430 consensus_queue.cc:1035] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:58:00.184252 7882 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:58:00.184350 7882 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:58:00.184927 7882 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9: Sending DeleteTablet for 3 replicas of tablet 35405a51afe94882ad979d12e6f8e795
I20260924 02:58:00.185160 8160 tablet_service.cc:1514] Processing DeleteTablet for tablet 35405a51afe94882ad979d12e6f8e795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:58:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51970
I20260924 02:58:00.185250 8414 tablet_replica.cc:330] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0: stopping tablet replica
I20260924 02:58:00.185236 8020 tablet_service.cc:1514] Processing DeleteTablet for tablet 35405a51afe94882ad979d12e6f8e795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:58:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55534
I20260924 02:58:00.185295 8414 raft_consensus.cc:2229] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:58:00.185329 8414 raft_consensus.cc:2258] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:58:00.185345 8413 tablet_replica.cc:330] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a: stopping tablet replica
I20260924 02:58:00.185405 8413 raft_consensus.cc:2229] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:58:00.185451 8413 raft_consensus.cc:2258] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:58:00.185513 8300 tablet_service.cc:1514] Processing DeleteTablet for tablet 35405a51afe94882ad979d12e6f8e795 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:58:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34628
I20260924 02:58:00.185482 7882 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53500:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260924 02:58:00.185583 8415 tablet_replica.cc:330] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1: stopping tablet replica
W20260924 02:58:00.185608 7882 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:58:00.185624 8415 raft_consensus.cc:2229] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [term 1 LEADER]: Raft consensus shutting down.
I20260924 02:58:00.185704 8415 raft_consensus.cc:2258] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:58:00.185740 8413 ts_tablet_manager.cc:1905] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:58:00.185985 8415 ts_tablet_manager.cc:1905] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:58:00.187175 8413 ts_tablet_manager.cc:1918] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:58:00.187225 8413 log.cc:1196] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-0/wal/wals/35405a51afe94882ad979d12e6f8e795
I20260924 02:58:00.187384 8413 ts_tablet_manager.cc:1939] T 35405a51afe94882ad979d12e6f8e795 P 71b2ecc8215f4e608fd14e6f07625b7a: Deleting consensus metadata
I20260924 02:58:00.187388 8415 ts_tablet_manager.cc:1918] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:58:00.187435 8415 log.cc:1196] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/35405a51afe94882ad979d12e6f8e795
I20260924 02:58:00.187631 8415 ts_tablet_manager.cc:1939] T 35405a51afe94882ad979d12e6f8e795 P 47faee667646449abd37257851cb71a1: Deleting consensus metadata
I20260924 02:58:00.187650 7869 catalog_manager.cc:4841] TS 71b2ecc8215f4e608fd14e6f07625b7a (127.0.98.1:37615): tablet 35405a51afe94882ad979d12e6f8e795 (table test-table [id=58455af841d444f1a848fa739cba3b80]) successfully deleted
I20260924 02:58:00.188000 7869 catalog_manager.cc:4841] TS 47faee667646449abd37257851cb71a1 (127.0.98.3:46189): tablet 35405a51afe94882ad979d12e6f8e795 (table test-table [id=58455af841d444f1a848fa739cba3b80]) successfully deleted
I20260924 02:58:00.188294 8414 ts_tablet_manager.cc:1905] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:58:00.188727 8020 tablet_service.cc:1467] Processing CreateTablet for tablet 6323adecc7bd4e74baed5faee7d7f0ce (DEFAULT_TABLE table=test-table [id=25383f1011f048dc9036a64aa59f8aa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:58:00.188833 8020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6323adecc7bd4e74baed5faee7d7f0ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:58:00.189615 8414 ts_tablet_manager.cc:1918] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:58:00.189674 8414 log.cc:1196] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/35405a51afe94882ad979d12e6f8e795
I20260924 02:58:00.189745 8391 tablet_bootstrap.cc:492] T 6323adecc7bd4e74baed5faee7d7f0ce P 71b2ecc8215f4e608fd14e6f07625b7a: Bootstrap starting.
I20260924 02:58:00.189857 8414 ts_tablet_manager.cc:1939] T 35405a51afe94882ad979d12e6f8e795 P b738799e612c4554a4ad08102c417de0: Deleting consensus metadata
I20260924 02:58:00.190125 8160 tablet_service.cc:1467] Processing CreateTablet for tablet 6323adecc7bd4e74baed5faee7d7f0ce (DEFAULT_TABLE table=test-table [id=25383f1011f048dc9036a64aa59f8aa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:58:00.190224 8160 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6323adecc7bd4e74baed5faee7d7f0ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:58:00.190315 8391 tablet_bootstrap.cc:654] T 6323adecc7bd4e74baed5faee7d7f0ce P 71b2ecc8215f4e608fd14e6f07625b7a: Neither blocks nor log segments found. Creating new log.
I20260924 02:58:00.190658 8300 tablet_service.cc:1467] Processing CreateTablet for tablet 6323adecc7bd4e74baed5faee7d7f0ce (DEFAULT_TABLE table=test-table [id=25383f1011f048dc9036a64aa59f8aa3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:58:00.190748 8300 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6323adecc7bd4e74baed5faee7d7f0ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:58:00.191125 8392 tablet_bootstrap.cc:492] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0: Bootstrap starting.
I20260924 02:58:00.191133 7870 catalog_manager.cc:4841] TS b738799e612c4554a4ad08102c417de0 (127.0.98.2:46053): tablet 35405a51afe94882ad979d12e6f8e795 (table test-table [id=58455af841d444f1a848fa739cba3b80]) successfully deleted
I20260924 02:58:00.191431 8392 tablet_bootstrap.cc:654] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0: Neither blocks nor log segments found. Creating new log.
I20260924 02:58:00.191859 8395 tablet_bootstrap.cc:492] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1: Bootstrap starting.
I20260924 02:58:00.191879 8391 tablet_bootstrap.cc:492] T 6323adecc7bd4e74baed5faee7d7f0ce P 71b2ecc8215f4e608fd14e6f07625b7a: No bootstrap required, opened a new log
I20260924 02:58:00.191911 8391 ts_tablet_manager.cc:1397] T 6323adecc7bd4e74baed5faee7d7f0ce P 71b2ecc8215f4e608fd14e6f07625b7a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:58:00.192049 8391 raft_consensus.cc:348] T 6323adecc7bd4e74baed5faee7d7f0ce P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:58:00.192108 8391 raft_consensus.cc:374] T 6323adecc7bd4e74baed5faee7d7f0ce P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:58:00.192126 8391 raft_consensus.cc:729] T 6323adecc7bd4e74baed5faee7d7f0ce P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 71b2ecc8215f4e608fd14e6f07625b7a, State: Initialized, Role: FOLLOWER
I20260924 02:58:00.192165 8391 consensus_queue.cc:260] T 6323adecc7bd4e74baed5faee7d7f0ce P 71b2ecc8215f4e608fd14e6f07625b7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:58:00.192209 8395 tablet_bootstrap.cc:654] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1: Neither blocks nor log segments found. Creating new log.
I20260924 02:58:00.192251 8391 ts_tablet_manager.cc:1428] T 6323adecc7bd4e74baed5faee7d7f0ce P 71b2ecc8215f4e608fd14e6f07625b7a: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260924 02:58:00.192592 8392 tablet_bootstrap.cc:492] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0: No bootstrap required, opened a new log
I20260924 02:58:00.192633 8392 ts_tablet_manager.cc:1397] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:58:00.192725 8392 raft_consensus.cc:348] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:58:00.192781 8392 raft_consensus.cc:374] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:58:00.192798 8392 raft_consensus.cc:729] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b738799e612c4554a4ad08102c417de0, State: Initialized, Role: FOLLOWER
I20260924 02:58:00.192833 8392 consensus_queue.cc:260] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:58:00.192924 8392 ts_tablet_manager.cc:1428] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:00.193370 8395 tablet_bootstrap.cc:492] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1: No bootstrap required, opened a new log
I20260924 02:58:00.193418 8395 ts_tablet_manager.cc:1397] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:58:00.193519 8395 raft_consensus.cc:348] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:58:00.193578 8395 raft_consensus.cc:374] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:58:00.193595 8395 raft_consensus.cc:729] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47faee667646449abd37257851cb71a1, State: Initialized, Role: FOLLOWER
I20260924 02:58:00.193631 8395 consensus_queue.cc:260] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:58:00.193711 8395 ts_tablet_manager.cc:1428] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:00.220050 8442 raft_consensus.cc:482] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:58:00.220152 8442 raft_consensus.cc:504] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:58:00.220302 8442 leader_election.cc:288] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 71b2ecc8215f4e608fd14e6f07625b7a (127.0.98.1:37615), 47faee667646449abd37257851cb71a1 (127.0.98.3:46189)
I20260924 02:58:00.220497 8320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6323adecc7bd4e74baed5faee7d7f0ce" candidate_uuid: "b738799e612c4554a4ad08102c417de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47faee667646449abd37257851cb71a1" is_pre_election: true
I20260924 02:58:00.220497 8040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6323adecc7bd4e74baed5faee7d7f0ce" candidate_uuid: "b738799e612c4554a4ad08102c417de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" is_pre_election: true
I20260924 02:58:00.220602 8320 raft_consensus.cc:2454] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b738799e612c4554a4ad08102c417de0 in term 0.
I20260924 02:58:00.220602 8040 raft_consensus.cc:2454] T 6323adecc7bd4e74baed5faee7d7f0ce P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b738799e612c4554a4ad08102c417de0 in term 0.
I20260924 02:58:00.220736 8113 leader_election.cc:302] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47faee667646449abd37257851cb71a1, b738799e612c4554a4ad08102c417de0; no voters:
I20260924 02:58:00.220842 8442 raft_consensus.cc:2781] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:58:00.220883 8442 raft_consensus.cc:482] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:58:00.220901 8442 raft_consensus.cc:3037] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:58:00.221566 8442 raft_consensus.cc:504] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:58:00.221668 8442 leader_election.cc:288] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [CANDIDATE]: Term 1 election: Requested vote from peers 71b2ecc8215f4e608fd14e6f07625b7a (127.0.98.1:37615), 47faee667646449abd37257851cb71a1 (127.0.98.3:46189)
I20260924 02:58:00.221861 8040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6323adecc7bd4e74baed5faee7d7f0ce" candidate_uuid: "b738799e612c4554a4ad08102c417de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "71b2ecc8215f4e608fd14e6f07625b7a"
I20260924 02:58:00.221884 8320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6323adecc7bd4e74baed5faee7d7f0ce" candidate_uuid: "b738799e612c4554a4ad08102c417de0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47faee667646449abd37257851cb71a1"
I20260924 02:58:00.221947 8320 raft_consensus.cc:3037] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:58:00.221947 8040 raft_consensus.cc:3037] T 6323adecc7bd4e74baed5faee7d7f0ce P 71b2ecc8215f4e608fd14e6f07625b7a [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:58:00.222632 8040 raft_consensus.cc:2454] T 6323adecc7bd4e74baed5faee7d7f0ce P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b738799e612c4554a4ad08102c417de0 in term 1.
I20260924 02:58:00.222632 8320 raft_consensus.cc:2454] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b738799e612c4554a4ad08102c417de0 in term 1.
I20260924 02:58:00.222782 8113 leader_election.cc:302] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 71b2ecc8215f4e608fd14e6f07625b7a, b738799e612c4554a4ad08102c417de0; no voters:
I20260924 02:58:00.222911 8442 raft_consensus.cc:2781] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:58:00.222961 8442 raft_consensus.cc:686] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [term 1 LEADER]: Becoming Leader. State: Replica: b738799e612c4554a4ad08102c417de0, State: Running, Role: LEADER
I20260924 02:58:00.223009 8442 consensus_queue.cc:237] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } }
I20260924 02:58:00.223516 7882 catalog_manager.cc:5495] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 reported cstate change: term changed from 0 to 1, leader changed from <none> to b738799e612c4554a4ad08102c417de0 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "b738799e612c4554a4ad08102c417de0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b738799e612c4554a4ad08102c417de0" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46053 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 } health_report { overall_health: UNKNOWN } } }
I20260924 02:58:00.231295 8387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:00.228880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.0.98.2:46053)
0924 02:58:00.228924 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:00.228926 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:00.228953 (+ 27us) server_negotiation.cc:407] Connection header received
0924 02:58:00.229021 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:00.229021 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:00.229032 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:00.229075 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:58:00.229178 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:00.229633 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:00.230671 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:00.230856 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:00.230913 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:00.231123 (+ 210us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:58:00.231148 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:58:00.231178 (+ 30us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:00.231228 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 02:58:00.231242 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 02:58:00.233229 8040 raft_consensus.cc:1261] T 6323adecc7bd4e74baed5faee7d7f0ce P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Refusing update from remote peer b738799e612c4554a4ad08102c417de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:58:00.233237 8320 raft_consensus.cc:1261] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1 [term 1 FOLLOWER]: Refusing update from remote peer b738799e612c4554a4ad08102c417de0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:58:00.233423 8442 consensus_queue.cc:1035] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47faee667646449abd37257851cb71a1" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 46189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:58:00.233552 8401 consensus_queue.cc:1035] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "71b2ecc8215f4e608fd14e6f07625b7a" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:58:00.244149 7883 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:58:00.244331 7883 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53500:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:58:00.244937 7883 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6858ebb8fa404b13a1417e4b3e7c52b9: Sending DeleteTablet for 3 replicas of tablet 6323adecc7bd4e74baed5faee7d7f0ce
I20260924 02:58:00.245154 8160 tablet_service.cc:1514] Processing DeleteTablet for tablet 6323adecc7bd4e74baed5faee7d7f0ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:58:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51970
I20260924 02:58:00.245211 8300 tablet_service.cc:1514] Processing DeleteTablet for tablet 6323adecc7bd4e74baed5faee7d7f0ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:58:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34628
I20260924 02:58:00.245267 8020 tablet_service.cc:1514] Processing DeleteTablet for tablet 6323adecc7bd4e74baed5faee7d7f0ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:58:00 UTC) from {username='oryx', principal='oryx/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:55534
I20260924 02:58:00.245352 8414 tablet_replica.cc:330] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0: stopping tablet replica
I20260924 02:58:00.245402 8414 raft_consensus.cc:2229] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [term 1 LEADER]: Raft consensus shutting down.
I20260924 02:58:00.245464 8414 raft_consensus.cc:2258] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:58:00.245532 8415 tablet_replica.cc:330] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1: stopping tablet replica
I20260924 02:58:00.245580 8415 raft_consensus.cc:2229] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:58:00.245563 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 7955
I20260924 02:58:00.245610 8415 raft_consensus.cc:2258] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:58:00.245671 8413 tablet_replica.cc:330] T 6323adecc7bd4e74baed5faee7d7f0ce P 71b2ecc8215f4e608fd14e6f07625b7a: stopping tablet replica
I20260924 02:58:00.245718 8413 raft_consensus.cc:2229] T 6323adecc7bd4e74baed5faee7d7f0ce P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:58:00.245745 8413 raft_consensus.cc:2258] T 6323adecc7bd4e74baed5faee7d7f0ce P 71b2ecc8215f4e608fd14e6f07625b7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:58:00.246243 8415 ts_tablet_manager.cc:1905] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:58:00.247666 8415 ts_tablet_manager.cc:1918] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 02:58:00.247723 8415 log.cc:1196] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/6323adecc7bd4e74baed5faee7d7f0ce
I20260924 02:58:00.247900 8415 ts_tablet_manager.cc:1939] T 6323adecc7bd4e74baed5faee7d7f0ce P 47faee667646449abd37257851cb71a1: Deleting consensus metadata
I20260924 02:58:00.248152 8414 ts_tablet_manager.cc:1905] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:58:00.249307 8414 ts_tablet_manager.cc:1918] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260924 02:58:00.249359 8414 log.cc:1196] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/6323adecc7bd4e74baed5faee7d7f0ce
I20260924 02:58:00.249383 7869 catalog_manager.cc:4841] TS 47faee667646449abd37257851cb71a1 (127.0.98.3:46189): tablet 6323adecc7bd4e74baed5faee7d7f0ce (table test-table [id=25383f1011f048dc9036a64aa59f8aa3]) successfully deleted
I20260924 02:58:00.249537 8414 ts_tablet_manager.cc:1939] T 6323adecc7bd4e74baed5faee7d7f0ce P b738799e612c4554a4ad08102c417de0: Deleting consensus metadata
I20260924 02:58:00.249800 7870 catalog_manager.cc:4841] TS b738799e612c4554a4ad08102c417de0 (127.0.98.2:46053): tablet 6323adecc7bd4e74baed5faee7d7f0ce (table test-table [id=25383f1011f048dc9036a64aa59f8aa3]) successfully deleted
W20260924 02:58:00.251968 7869 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37615 (error 108)
I20260924 02:58:00.252168 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 8095
I20260924 02:58:00.252949 8474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:00.252296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37615 (local address 127.0.0.1:55540)
0924 02:58:00.252873 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:00.252917 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37615: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":37,"threads_started":1}
W20260924 02:58:00.253033 7869 catalog_manager.cc:4584] TS 71b2ecc8215f4e608fd14e6f07625b7a (127.0.98.1:37615): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6323adecc7bd4e74baed5faee7d7f0ce: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37615: connect: Connection refused (error 111)
I20260924 02:58:00.260273 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 8235
I20260924 02:58:00.265134 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 7851
2026-09-24T02:58:00Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2379 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:58:00 dist-test-slave-sb6p krb5kdc[8477](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:58:00 dist-test-slave-sb6p krb5kdc[8477](info): set up 2 sockets
Sep 24 02:58:00 dist-test-slave-sb6p krb5kdc[8477](info): commencing operation
krb5kdc: starting...
W20260924 02:58:01.206735 392 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:58:01 dist-test-slave-sb6p krb5kdc[8477](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:58:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:58:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 02:58:01.315475 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35043
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33017
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35043
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:58:01.393018 8490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:01.393222 8490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:01.393250 8490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:01.394426 8490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:58:01.394465 8490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:01.394482 8490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:01.394498 8490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:58:01.394531 8490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:58:01.396031 8490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33017
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35043
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35043
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:01.396263 8490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:01.396453 8490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 02:58:01.401054 8490 server_base.cc:1034] running on GCE node
W20260924 02:58:01.401026 8496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:01.401078 8498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:01.401026 8495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:01.401568 8490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:01.401831 8490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:01.402962 8490 hybrid_clock.cc:648] HybridClock initialized: now 1790218681402951 us; error 30 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-24T02:58:01Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1148 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:58:01 dist-test-slave-sb6p krb5kdc[8503](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:58:01 dist-test-slave-sb6p krb5kdc[8503](info): set up 2 sockets
Sep 24 02:58:01 dist-test-slave-sb6p krb5kdc[8503](info): commencing operation
krb5kdc: starting...
W20260924 02:58:02.366760 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:58:02 dist-test-slave-sb6p krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:58:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:58:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 02:58:02.481400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38311
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:58:02.568373 8516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:02.568573 8516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:02.568593 8516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:02.569649 8516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:58:02.569681 8516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:02.569693 8516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:02.569705 8516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:58:02.569715 8516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:58:02.571112 8516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:02.571292 8516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:02.571465 8516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:02.575932 8524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:02.575941 8522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:02.576099 8516 server_base.cc:1034] running on GCE node
W20260924 02:58:02.575950 8521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:02.576391 8516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:02.576645 8516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:02.577780 8516 hybrid_clock.cc:648] HybridClock initialized: now 1790218682577764 us; error 26 us; skew 500 ppm
Sep 24 02:58:02 dist-test-slave-sb6p krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:58:02.580060 8516 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:58:02.580425 8516 webserver.cc:460] Webserver started at http://127.0.98.62:39109/ using document root <none> and password file <none>
I20260924 02:58:02.580587 8516 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:02.580624 8516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:02.580709 8516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:58:02.581672 8516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "ae36ded474404f8bb191ee20bc69b176"
format_stamp: "Formatted at 2026-09-24 02:58:02 on dist-test-slave-sb6p"
server_key: "993828f233686fe916265f03a6f4e3d1"
server_key_iv: "717cc38569b8089e36b6bb5ae7315543"
server_key_version: "encryptionkey@0"
I20260924 02:58:02.581959 8516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "ae36ded474404f8bb191ee20bc69b176"
format_stamp: "Formatted at 2026-09-24 02:58:02 on dist-test-slave-sb6p"
server_key: "993828f233686fe916265f03a6f4e3d1"
server_key_iv: "717cc38569b8089e36b6bb5ae7315543"
server_key_version: "encryptionkey@0"
I20260924 02:58:02.583132 8516 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 02:58:02.583721 8531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:02.583866 8516 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:58:02.583943 8516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "ae36ded474404f8bb191ee20bc69b176"
format_stamp: "Formatted at 2026-09-24 02:58:02 on dist-test-slave-sb6p"
server_key: "993828f233686fe916265f03a6f4e3d1"
server_key_iv: "717cc38569b8089e36b6bb5ae7315543"
server_key_version: "encryptionkey@0"
I20260924 02:58:02.583995 8516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:02.602066 8516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:02.603824 8516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:02.603929 8516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:02.607448 8516 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38311
I20260924 02:58:02.607489 8583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38311 every 8 connection(s)
I20260924 02:58:02.607921 8516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:58:02.608523 8584 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:58:02.610705 8584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176: Bootstrap starting.
I20260924 02:58:02.611232 8584 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176: Neither blocks nor log segments found. Creating new log.
I20260924 02:58:02.611523 8584 log.cc:824] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176: Log is configured to *not* fsync() on all Append() calls
I20260924 02:58:02.612109 8584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176: No bootstrap required, opened a new log
I20260924 02:58:02.613621 8584 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36ded474404f8bb191ee20bc69b176" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } }
I20260924 02:58:02.613754 8584 raft_consensus.cc:374] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:58:02.613785 8584 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae36ded474404f8bb191ee20bc69b176, State: Initialized, Role: FOLLOWER
I20260924 02:58:02.613919 8584 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36ded474404f8bb191ee20bc69b176" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } }
I20260924 02:58:02.613979 8584 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:58:02.614001 8584 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:58:02.614044 8584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:58:02.614913 8584 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36ded474404f8bb191ee20bc69b176" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } }
I20260924 02:58:02.615007 8584 leader_election.cc:302] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae36ded474404f8bb191ee20bc69b176; no voters:
I20260924 02:58:02.615156 8584 leader_election.cc:288] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:58:02.615195 8587 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:58:02.615329 8587 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [term 1 LEADER]: Becoming Leader. State: Replica: ae36ded474404f8bb191ee20bc69b176, State: Running, Role: LEADER
I20260924 02:58:02.615396 8584 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:58:02.615422 8587 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36ded474404f8bb191ee20bc69b176" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } }
I20260924 02:58:02.615738 8589 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae36ded474404f8bb191ee20bc69b176. Latest consensus state: current_term: 1 leader_uuid: "ae36ded474404f8bb191ee20bc69b176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36ded474404f8bb191ee20bc69b176" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } } }
I20260924 02:58:02.615837 8589 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [sys.catalog]: This master's current role is: LEADER
I20260924 02:58:02.615734 8588 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ae36ded474404f8bb191ee20bc69b176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36ded474404f8bb191ee20bc69b176" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } } }
I20260924 02:58:02.616003 8588 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [sys.catalog]: This master's current role is: LEADER
I20260924 02:58:02.616096 8595 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:58:02.616612 8595 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:58:02.616664 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 8516
I20260924 02:58:02.616781 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:58:02.616958 392 external_mini_cluster.cc:1428] Setting key b31202d8194245c33c0c75298cdec9fb
I20260924 02:58:02.618095 8595 catalog_manager.cc:1347] Generated new cluster ID: 540cd4c5af334a76ab9ede412a924bd5
I20260924 02:58:02.618158 8595 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 02:58:02 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218682, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:58:02.628233 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:02.617561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57588 (local address 127.0.98.62:38311)
0924 02:58:02.617817 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:02.617826 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:02.617856 (+ 30us) server_negotiation.cc:407] Connection header received
0924 02:58:02.617976 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:02.617982 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:02.618045 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:02.618897 (+ 852us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:02.618993 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:02.620352 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:02.621462 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:02.622066 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:02.622412 (+ 346us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:02.624864 (+ 2452us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:02.624868 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:02.624868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:58:02.624881 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:02.626702 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:02.627080 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:02.627082 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:02.627082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:02.627180 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:02.627359 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:02.627359 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:02.627360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:02.627639 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:02.627751 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:02.627865 (+ 114us) server_negotiation.cc:299] Negotiation successful
0924 02:58:02.628004 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":69,"threads_started":1}
I20260924 02:58:02.634785 8595 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:58:02.635510 8595 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:58:02.641755 8595 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176: Generated new TSK 0
I20260924 02:58:02.641939 8595 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 02:58:02.667689 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:58:02.745448 8610 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:02.745642 8610 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:02.745671 8610 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:02.746819 8610 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:02.746855 8610 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:02.746889 8610 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:58:02.748328 8610 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8610
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:02.748548 8610 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:02.748751 8610 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:02.753345 8618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:02.753352 8615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:02.753589 8610 server_base.cc:1034] running on GCE node
W20260924 02:58:02.753618 8616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:02.753820 8610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:02.754040 8610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:02.755182 8610 hybrid_clock.cc:648] HybridClock initialized: now 1790218682755168 us; error 31 us; skew 500 ppm
Sep 24 02:58:02 dist-test-slave-sb6p krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:58:02.757579 8610 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 02:58:02.757946 8610 webserver.cc:460] Webserver started at http://127.0.98.1:41277/ using document root <none> and password file <none>
I20260924 02:58:02.758106 8610 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:02.758142 8610 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:02.758229 8610 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:58:02.759195 8610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "709e2cc577454a808a829fb9be7eb9a9"
format_stamp: "Formatted at 2026-09-24 02:58:02 on dist-test-slave-sb6p"
server_key: "e0928cfbe6383a189bc5db6427ab41f3"
server_key_iv: "78e69eb36c7b0dd93af17c4ba6b315b8"
server_key_version: "encryptionkey@0"
I20260924 02:58:02.759487 8610 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "709e2cc577454a808a829fb9be7eb9a9"
format_stamp: "Formatted at 2026-09-24 02:58:02 on dist-test-slave-sb6p"
server_key: "e0928cfbe6383a189bc5db6427ab41f3"
server_key_iv: "78e69eb36c7b0dd93af17c4ba6b315b8"
server_key_version: "encryptionkey@0"
I20260924 02:58:02.760586 8610 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:58:02.761180 8625 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:02.761324 8610 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:58:02.761399 8610 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "709e2cc577454a808a829fb9be7eb9a9"
format_stamp: "Formatted at 2026-09-24 02:58:02 on dist-test-slave-sb6p"
server_key: "e0928cfbe6383a189bc5db6427ab41f3"
server_key_iv: "78e69eb36c7b0dd93af17c4ba6b315b8"
server_key_version: "encryptionkey@0"
I20260924 02:58:02.761456 8610 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:02.776327 8610 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:02.778033 8610 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:02.778136 8610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:02.778301 8610 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:58:02.778580 8610 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:58:02.778611 8610 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:02.778631 8610 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:58:02.778647 8610 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:02.783835 8610 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33783
I20260924 02:58:02.783874 8738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33783 every 8 connection(s)
I20260924 02:58:02.784178 8610 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 02:58:02 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:58:02.792304 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 8610
I20260924 02:58:02.792404 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:58:02.792578 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:02.784823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:58553 (local address 127.0.98.62:38311)
0924 02:58:02.784874 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:02.784876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:02.785063 (+ 187us) server_negotiation.cc:407] Connection header received
0924 02:58:02.785942 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:02.785943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:02.785956 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:02.786014 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:02.786660 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:02.787196 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:02.788147 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:02.788361 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:02.788407 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:02.790318 (+ 1911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:02.790319 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:02.790319 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:58:02.790324 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:02.791467 (+ 1143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:02.791788 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:02.791788 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:02.791789 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:02.791832 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:02.792003 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:02.792003 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:02.792003 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:02.792091 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:02.792133 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:02.792438 (+ 305us) server_negotiation.cc:299] Negotiation successful
0924 02:58:02.792516 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 02:58:02.792629 392 external_mini_cluster.cc:1428] Setting key cab8a6d1cc121032b1eff14e0d816bd9
I20260924 02:58:02.792778 8741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:02.784793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:58553)
0924 02:58:02.784943 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:02.784984 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:02.785856 (+ 872us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:02.786080 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:02.786083 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:02.786340 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:02.786591 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:02.786596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:02.787258 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:02.787259 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:02.788083 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:02.788085 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:02.788422 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:02.788423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:02.788547 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:02.788994 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:02.789003 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:02.790244 (+ 1241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:58:02.791528 (+ 1284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:58:02.791533 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:58:02.791535 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:58:02.791732 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:58:02.791874 (+ 142us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:58:02.791875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:58:02.791875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:58:02.791961 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:58:02.792137 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:58:02.792142 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:58:02.792293 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 02:58:02.792404 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 02:58:02.792566 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":68,"threads_started":1}
I20260924 02:58:02.793290 8739 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38311
I20260924 02:58:02.793390 8739 heartbeater.cc:464] Registering TS with master...
I20260924 02:58:02.793587 8739 heartbeater.cc:511] Master 127.0.98.62:38311 requested a full tablet report, sending...
I20260924 02:58:02.794098 8548 ts_manager.cc:194] Registered new tserver with Master: 709e2cc577454a808a829fb9be7eb9a9 (127.0.98.1:33783)
I20260924 02:58:02.794862 8548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:58553
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 02:58:02.815125 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:58:02.893123 8744 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:02.893322 8744 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:02.893343 8744 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:02.894469 8744 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:02.894510 8744 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:02.894562 8744 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:58:02.896020 8744 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8744
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:02.896219 8744 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:02.896394 8744 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:02.900997 8749 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:02.901000 8752 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:02.901065 8750 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:02.901031 8744 server_base.cc:1034] running on GCE node
I20260924 02:58:02.901393 8744 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:02.901621 8744 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:02.902752 8744 hybrid_clock.cc:648] HybridClock initialized: now 1790218682902736 us; error 27 us; skew 500 ppm
Sep 24 02:58:02 dist-test-slave-sb6p krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:58:02.904981 8744 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 02:58:02.905349 8744 webserver.cc:460] Webserver started at http://127.0.98.2:37361/ using document root <none> and password file <none>
I20260924 02:58:02.905531 8744 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:02.905577 8744 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:02.905705 8744 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:58:02.906736 8744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "6e9e03f9b4c54ab5b17411cb2307460e"
format_stamp: "Formatted at 2026-09-24 02:58:02 on dist-test-slave-sb6p"
server_key: "06616bae05f9decf7ccd013f239f6de4"
server_key_iv: "241a9b4cf1a5879aa64fbeaaeff3da1c"
server_key_version: "encryptionkey@0"
I20260924 02:58:02.907058 8744 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6e9e03f9b4c54ab5b17411cb2307460e"
format_stamp: "Formatted at 2026-09-24 02:58:02 on dist-test-slave-sb6p"
server_key: "06616bae05f9decf7ccd013f239f6de4"
server_key_iv: "241a9b4cf1a5879aa64fbeaaeff3da1c"
server_key_version: "encryptionkey@0"
I20260924 02:58:02.908200 8744 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:58:02.908809 8759 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:02.908967 8744 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:58:02.909047 8744 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "6e9e03f9b4c54ab5b17411cb2307460e"
format_stamp: "Formatted at 2026-09-24 02:58:02 on dist-test-slave-sb6p"
server_key: "06616bae05f9decf7ccd013f239f6de4"
server_key_iv: "241a9b4cf1a5879aa64fbeaaeff3da1c"
server_key_version: "encryptionkey@0"
I20260924 02:58:02.909106 8744 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:02.919510 8744 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:02.921114 8744 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:02.921237 8744 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:02.921435 8744 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:58:02.921705 8744 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:58:02.921736 8744 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:02.921763 8744 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:58:02.921787 8744 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:02.927171 8744 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42337
I20260924 02:58:02.927219 8872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42337 every 8 connection(s)
I20260924 02:58:02.927505 8744 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 02:58:02.929329 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 8744
I20260924 02:58:02.929456 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:58:02.929646 392 external_mini_cluster.cc:1428] Setting key 2c4b41842fd3f4e556e72b1509b547ce
Sep 24 02:58:02 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:58:02.937220 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:02.928159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58853 (local address 127.0.98.62:38311)
0924 02:58:02.928218 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:02.928220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:02.928379 (+ 159us) server_negotiation.cc:407] Connection header received
0924 02:58:02.929315 (+ 936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:02.929317 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:02.929329 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:02.929388 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:02.930133 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:02.930770 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:02.932030 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:02.932220 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:02.932274 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:02.934546 (+ 2272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:02.934548 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:02.934548 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:58:02.934556 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:02.935801 (+ 1245us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:02.936253 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:02.936253 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:02.936254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:02.936358 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:02.936622 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:02.936623 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:02.936624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:02.936736 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:02.936793 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:02.937096 (+ 303us) server_negotiation.cc:299] Negotiation successful
0924 02:58:02.937157 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 02:58:02.937467 8875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:02.928126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:58853)
0924 02:58:02.928265 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:02.928304 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:02.929180 (+ 876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:02.929476 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:02.929480 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:02.929815 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:02.930050 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:02.930055 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:02.930852 (+ 797us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:02.930854 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:02.931955 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:02.931958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:02.932285 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:02.932286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:02.932400 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:02.932871 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:02.932879 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:02.934434 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:58:02.935885 (+ 1451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:58:02.935892 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:58:02.935894 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:58:02.936182 (+ 288us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:58:02.936417 (+ 235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:58:02.936418 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:58:02.936418 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:58:02.936542 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:58:02.936801 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:58:02.936806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:58:02.936967 (+ 161us) client_negotiation.cc:769] Sending connection context
0924 02:58:02.937089 (+ 122us) client_negotiation.cc:240] Negotiation successful
0924 02:58:02.937253 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 02:58:02.937947 8873 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38311
I20260924 02:58:02.938031 8873 heartbeater.cc:464] Registering TS with master...
I20260924 02:58:02.938189 8873 heartbeater.cc:511] Master 127.0.98.62:38311 requested a full tablet report, sending...
I20260924 02:58:02.938500 8548 ts_manager.cc:194] Registered new tserver with Master: 6e9e03f9b4c54ab5b17411cb2307460e (127.0.98.2:42337)
I20260924 02:58:02.938897 8548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:58853
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 02:58:02.953514 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:58:03.031448 8878 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:03.031641 8878 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:03.031662 8878 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:03.032739 8878 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:03.032781 8878 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:03.032806 8878 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:58:03.034228 8878 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8878
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:03.034412 8878 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:03.034633 8878 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:03.039299 8883 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:03.039290 8884 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:03.039290 8886 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:03.039505 8878 server_base.cc:1034] running on GCE node
I20260924 02:58:03.039695 8878 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:03.039888 8878 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:03.041009 8878 hybrid_clock.cc:648] HybridClock initialized: now 1790218683041001 us; error 25 us; skew 500 ppm
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:58:03.043319 8878 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 02:58:03.043720 8878 webserver.cc:460] Webserver started at http://127.0.98.3:39633/ using document root <none> and password file <none>
I20260924 02:58:03.043905 8878 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:03.043954 8878 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:03.044075 8878 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:58:03.045073 8878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "a651cf9b65384056acbd0c4cf6738231"
format_stamp: "Formatted at 2026-09-24 02:58:03 on dist-test-slave-sb6p"
server_key: "4cad209cece93eb3f984d70f32b45f1e"
server_key_iv: "b5e182212bcb492e6f3255ed451a1b26"
server_key_version: "encryptionkey@0"
I20260924 02:58:03.045375 8878 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "a651cf9b65384056acbd0c4cf6738231"
format_stamp: "Formatted at 2026-09-24 02:58:03 on dist-test-slave-sb6p"
server_key: "4cad209cece93eb3f984d70f32b45f1e"
server_key_iv: "b5e182212bcb492e6f3255ed451a1b26"
server_key_version: "encryptionkey@0"
I20260924 02:58:03.046595 8878 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 02:58:03.047271 8893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:03.047408 8878 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:58:03.047485 8878 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "a651cf9b65384056acbd0c4cf6738231"
format_stamp: "Formatted at 2026-09-24 02:58:03 on dist-test-slave-sb6p"
server_key: "4cad209cece93eb3f984d70f32b45f1e"
server_key_iv: "b5e182212bcb492e6f3255ed451a1b26"
server_key_version: "encryptionkey@0"
I20260924 02:58:03.047552 8878 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:03.055853 8878 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:03.057454 8878 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:03.057570 8878 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:03.057765 8878 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:58:03.058034 8878 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:58:03.058065 8878 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:03.058086 8878 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:58:03.058110 8878 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:03.063381 8878 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44431
I20260924 02:58:03.063419 9006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44431 every 8 connection(s)
I20260924 02:58:03.063690 8878 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 02:58:03.067467 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 8878
I20260924 02:58:03.067552 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:58:03.067706 392 external_mini_cluster.cc:1428] Setting key 66870ab6c6c31499d3aefd25189e7534
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:58:03.072237 8606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:03.064335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33507 (local address 127.0.98.62:38311)
0924 02:58:03.064385 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:03.064387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:03.064576 (+ 189us) server_negotiation.cc:407] Connection header received
0924 02:58:03.065520 (+ 944us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:03.065521 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:03.065534 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:03.065585 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:03.066182 (+ 597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.066742 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.067725 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.067956 (+ 231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.068012 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.069927 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:03.069928 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:03.069928 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:58:03.069935 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:03.071037 (+ 1102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:03.071379 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:03.071380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:03.071380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:03.071433 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:03.071647 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:03.071647 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:03.071648 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:03.071757 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:03.071811 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:03.072061 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 02:58:03.072141 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 02:58:03.072382 9009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:03.064301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:33507)
0924 02:58:03.064466 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:03.064500 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:03.065421 (+ 921us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:03.065652 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:03.065656 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:03.065902 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:03.066090 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.066094 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.066822 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.066823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.067649 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.067653 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.068026 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.068027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.068146 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.068603 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:03.068611 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:03.069843 (+ 1232us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:58:03.071110 (+ 1267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:58:03.071116 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:58:03.071118 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:58:03.071316 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:58:03.071498 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:58:03.071499 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:58:03.071499 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:58:03.071594 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:58:03.071808 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:58:03.071815 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:58:03.071948 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 02:58:03.072031 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 02:58:03.072176 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260924 02:58:03.072790 9007 heartbeater.cc:347] Connected to a master server at 127.0.98.62:38311
I20260924 02:58:03.072872 9007 heartbeater.cc:464] Registering TS with master...
I20260924 02:58:03.073022 9007 heartbeater.cc:511] Master 127.0.98.62:38311 requested a full tablet report, sending...
I20260924 02:58:03.073342 8548 ts_manager.cc:194] Registered new tserver with Master: a651cf9b65384056acbd0c4cf6738231 (127.0.98.3:44431)
I20260924 02:58:03.073660 8548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:33507
I20260924 02:58:03.079202 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/oryx_127.0.98.1.keytab.
Entry for principal oryx/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/oryx_127.0.98.1.keytab.
WARNING: no policy specified for oryx/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/oryx_127.0.98.2.keytab.
Entry for principal oryx/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/oryx_127.0.98.2.keytab.
WARNING: no policy specified for oryx/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/oryx_127.0.98.3.keytab.
Entry for principal oryx/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/oryx_127.0.98.3.keytab.
I20260924 02:58:03.142977 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 8610
I20260924 02:58:03.146684 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 8744
I20260924 02:58:03.150223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 8878
I20260924 02:58:03.153714 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 8516
I20260924 02:58:03.157644 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38311
--webserver_interface=127.0.98.62
--webserver_port=39109
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:38311
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:58:03.234436 9016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:03.234668 9016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:03.234691 9016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:03.235802 9016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:58:03.235836 9016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:03.235849 9016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:03.235860 9016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:58:03.235870 9016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:58:03.237313 9016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:38311
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:38311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=39109
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:03.237499 9016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:03.237666 9016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:03.242007 9022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:03.242053 9016 server_base.cc:1034] running on GCE node
W20260924 02:58:03.242002 9021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:03.241981 9024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:03.242377 9016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:03.242660 9016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:03.243813 9016 hybrid_clock.cc:648] HybridClock initialized: now 1790218683243797 us; error 27 us; skew 500 ppm
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:58:03.246024 9016 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:58:03.246421 9016 webserver.cc:460] Webserver started at http://127.0.98.62:39109/ using document root <none> and password file <none>
I20260924 02:58:03.246665 9016 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:03.246727 9016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:03.247954 9016 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:58:03.248484 9031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:03.248624 9016 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:58:03.248713 9016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "ae36ded474404f8bb191ee20bc69b176"
format_stamp: "Formatted at 2026-09-24 02:58:02 on dist-test-slave-sb6p"
server_key: "993828f233686fe916265f03a6f4e3d1"
server_key_iv: "717cc38569b8089e36b6bb5ae7315543"
server_key_version: "encryptionkey@0"
I20260924 02:58:03.249037 9016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:03.268500 9016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:03.270130 9016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:03.270259 9016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:03.273962 9016 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:38311
I20260924 02:58:03.274004 9083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:38311 every 8 connection(s)
I20260924 02:58:03.274497 9016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:58:03.275720 9084 sys_catalog.cc:263] Verifying existing consensus state
I20260924 02:58:03.276341 9084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176: Bootstrap starting.
I20260924 02:58:03.281662 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 9016
I20260924 02:58:03.282008 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:33783
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:38311
--webserver_port=41277
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.1
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/oryx_127.0.98.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 02:58:03.284332 9084 log.cc:824] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176: Log is configured to *not* fsync() on all Append() calls
I20260924 02:58:03.286000 9084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 02:58:03.286245 9084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176: Bootstrap complete.
I20260924 02:58:03.288301 9084 raft_consensus.cc:348] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36ded474404f8bb191ee20bc69b176" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } }
I20260924 02:58:03.288525 9084 raft_consensus.cc:729] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae36ded474404f8bb191ee20bc69b176, State: Initialized, Role: FOLLOWER
I20260924 02:58:03.288666 9084 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36ded474404f8bb191ee20bc69b176" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } }
I20260924 02:58:03.288731 9084 raft_consensus.cc:388] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:58:03.288774 9084 raft_consensus.cc:482] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:58:03.288818 9084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [term 1 FOLLOWER]: Advancing to term 2
I20260924 02:58:03.289779 9084 raft_consensus.cc:504] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36ded474404f8bb191ee20bc69b176" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } }
I20260924 02:58:03.289866 9084 leader_election.cc:302] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ae36ded474404f8bb191ee20bc69b176; no voters:
I20260924 02:58:03.290014 9084 leader_election.cc:288] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 02:58:03.290069 9088 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 02:58:03.290174 9088 raft_consensus.cc:686] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [term 2 LEADER]: Becoming Leader. State: Replica: ae36ded474404f8bb191ee20bc69b176, State: Running, Role: LEADER
I20260924 02:58:03.290266 9084 sys_catalog.cc:564] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:58:03.290256 9088 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36ded474404f8bb191ee20bc69b176" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } }
I20260924 02:58:03.290504 9090 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "ae36ded474404f8bb191ee20bc69b176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36ded474404f8bb191ee20bc69b176" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } } }
I20260924 02:58:03.290614 9090 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [sys.catalog]: This master's current role is: LEADER
I20260924 02:58:03.290501 9088 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae36ded474404f8bb191ee20bc69b176. Latest consensus state: current_term: 2 leader_uuid: "ae36ded474404f8bb191ee20bc69b176" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ae36ded474404f8bb191ee20bc69b176" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 38311 } } }
I20260924 02:58:03.290792 9088 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176 [sys.catalog]: This master's current role is: LEADER
I20260924 02:58:03.291100 9093 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:58:03.291561 9093 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:58:03.291724 9093 catalog_manager.cc:1259] Loaded cluster ID: 540cd4c5af334a76ab9ede412a924bd5
I20260924 02:58:03.291760 9093 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:58:03.293215 9093 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:58:03.293387 9093 catalog_manager.cc:5879] T 00000000000000000000000000000000 P ae36ded474404f8bb191ee20bc69b176: Loaded TSK: 0
I20260924 02:58:03.293589 9093 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 02:58:03.361817 9086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:03.362013 9086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:03.362036 9086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:03.363224 9086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:03.363265 9086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:03.363291 9086 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:58:03.364674 9086 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/oryx_127.0.98.1.keytab
--principal=oryx/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:33783
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=41277
--tserver_master_addrs=127.0.98.62:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:03.364878 9086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:03.365092 9086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:03.369877 9110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:03.370033 9086 server_base.cc:1034] running on GCE node
W20260924 02:58:03.370015 9113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:03.369903 9111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:03.370405 9086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:03.370699 9086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:03.371873 9086 hybrid_clock.cc:648] HybridClock initialized: now 1790218683371860 us; error 27 us; skew 500 ppm
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218683, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:58:03.374825 9086 init.cc:377] Logged in from keytab as oryx/127.0.98.1@KRBTEST.COM (short username oryx)
I20260924 02:58:03.375334 9086 webserver.cc:460] Webserver started at http://127.0.98.1:41277/ using document root <none> and password file <none>
I20260924 02:58:03.375564 9086 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:03.375636 9086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:03.377257 9086 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:58:03.377849 9120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:03.377983 9086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:58:03.378067 9086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "709e2cc577454a808a829fb9be7eb9a9"
format_stamp: "Formatted at 2026-09-24 02:58:02 on dist-test-slave-sb6p"
server_key: "e0928cfbe6383a189bc5db6427ab41f3"
server_key_iv: "78e69eb36c7b0dd93af17c4ba6b315b8"
server_key_version: "encryptionkey@0"
I20260924 02:58:03.378379 9086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:03.394593 9086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:03.396586 9086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:03.396721 9086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:03.396935 9086 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:58:03.397222 9086 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:58:03.397253 9086 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:03.397274 9086 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:58:03.397286 9086 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:03.402643 9086 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:33783
I20260924 02:58:03.402693 9233 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:33783 every 8 connection(s)
I20260924 02:58:03.403014 9086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:58:03.406899 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 9086
I20260924 02:58:03.407163 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:42337
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:38311
--webserver_port=37361
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.2
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/oryx_127.0.98.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:03.412382 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:03.403702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:47035)
0924 02:58:03.403855 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:03.403898 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:03.404389 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:03.404902 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:03.404906 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:03.405592 (+ 686us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:03.405782 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.405787 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.407454 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.407455 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.408728 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.408731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.409438 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.409439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.409567 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.410076 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:03.410085 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:03.412176 (+ 2091us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":67,"threads_started":1}
W20260924 02:58:03.412602 9234 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38311 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38311: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
W20260924 02:58:03.412734 9237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:03.403774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47035 (local address 127.0.98.62:38311)
0924 02:58:03.403970 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:03.403979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:03.404013 (+ 34us) server_negotiation.cc:407] Connection header received
0924 02:58:03.404647 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:03.404654 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:03.404710 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:03.404847 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:03.405899 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.407331 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.408840 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.409350 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.409469 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.412314 (+ 2845us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47035: BlockingRecv error: recv got EOF from 127.0.98.1:47035 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:03.422838 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:03.412842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:46801)
0924 02:58:03.418605 (+ 5763us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:03.418620 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:03.418689 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:03.418820 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:03.418820 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:03.418973 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:03.419029 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.419031 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.419643 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.419644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.420157 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.420158 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.420481 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.420482 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.420545 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.420981 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:03.420987 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:03.422782 (+ 1795us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":5735}
W20260924 02:58:03.422983 9237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:03.412937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46801 (local address 127.0.98.62:38311)
0924 02:58:03.412974 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:03.412976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:03.418684 (+ 5708us) server_negotiation.cc:407] Connection header received
0924 02:58:03.418754 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:03.418755 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:03.418769 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:03.418812 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:03.419095 (+ 283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.419573 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.420228 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.420410 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.420463 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.422913 (+ 2450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46801: BlockingRecv error: recv got EOF from 127.0.98.1:46801 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:03.427075 9236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:03.423114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:47535)
0924 02:58:03.423192 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:03.423202 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:03.423238 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:03.423355 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:03.423356 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:03.423471 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:03.423502 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.423503 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.424076 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.424077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.424556 (+ 479us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.424557 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.424851 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.424851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.424900 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.425354 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:03.425357 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:03.427007 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":62}
W20260924 02:58:03.427191 9237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:03.423206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47535 (local address 127.0.98.62:38311)
0924 02:58:03.423241 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:03.423243 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:03.423250 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:58:03.423289 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:03.423290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:03.423300 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:03.423346 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:03.423561 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.424012 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.424622 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.424798 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.424848 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.427088 (+ 2240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:47535: BlockingRecv error: recv got EOF from 127.0.98.1:47535 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 02:58:03.427208 9234 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 02:58:03.490940 9238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:03.491153 9238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:03.491183 9238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:03.492486 9238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:03.492532 9238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:03.492568 9238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:58:03.494002 9238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/oryx_127.0.98.2.keytab
--principal=oryx/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:42337
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=37361
--tserver_master_addrs=127.0.98.62:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:03.494230 9238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:03.494453 9238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:03.499308 9243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:03.499326 9244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:03.499326 9246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:03.499567 9238 server_base.cc:1034] running on GCE node
I20260924 02:58:03.499735 9238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:03.499923 9238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:03.501055 9238 hybrid_clock.cc:648] HybridClock initialized: now 1790218683501038 us; error 27 us; skew 500 ppm
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218683, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:58:03.503427 9238 init.cc:377] Logged in from keytab as oryx/127.0.98.2@KRBTEST.COM (short username oryx)
I20260924 02:58:03.503832 9238 webserver.cc:460] Webserver started at http://127.0.98.2:37361/ using document root <none> and password file <none>
I20260924 02:58:03.504033 9238 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:03.504094 9238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:03.505404 9238 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:58:03.505936 9253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:03.506089 9238 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:58:03.506171 9238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "6e9e03f9b4c54ab5b17411cb2307460e"
format_stamp: "Formatted at 2026-09-24 02:58:02 on dist-test-slave-sb6p"
server_key: "06616bae05f9decf7ccd013f239f6de4"
server_key_iv: "241a9b4cf1a5879aa64fbeaaeff3da1c"
server_key_version: "encryptionkey@0"
I20260924 02:58:03.506538 9238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:03.523619 9238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:03.525682 9238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:03.525817 9238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:03.526041 9238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:58:03.526335 9238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:58:03.526368 9238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:03.526396 9238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:58:03.526420 9238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:03.531684 9238 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42337
I20260924 02:58:03.531725 9366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42337 every 8 connection(s)
I20260924 02:58:03.532063 9238 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 02:58:03.538712 9237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:03.532769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53329 (local address 127.0.98.62:38311)
0924 02:58:03.532819 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:03.532821 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:03.533031 (+ 210us) server_negotiation.cc:407] Connection header received
0924 02:58:03.533508 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:03.533509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:03.533521 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:03.533557 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:03.534418 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.534936 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.535882 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.536064 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.536108 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.538664 (+ 2556us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53329: BlockingRecv error: recv got EOF from 127.0.98.2:53329 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 02:58:03.538729 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:03.532767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:53329)
0924 02:58:03.532916 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:03.532952 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:03.533404 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:03.533621 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:03.533625 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:03.534167 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:03.534353 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.534357 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.535013 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.535014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.535823 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.535825 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.536115 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.536116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.536206 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.536648 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:03.536657 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:03.538586 (+ 1929us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
W20260924 02:58:03.538939 9367 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38311 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38311: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:03.542117 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 9238
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:03.542400 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:44431
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:38311
--webserver_port=39633
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:45353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.98.3
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/oryx_127.0.98.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 02:58:03.542589 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:03.539138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:40259)
0924 02:58:03.539178 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:03.539188 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:03.539218 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:03.539315 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:03.539316 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:03.539413 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:03.539456 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.539457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.540009 (+ 552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.540009 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.540478 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.540479 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.540735 (+ 256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.540735 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.540775 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.541074 (+ 299us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:03.541077 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:03.542514 (+ 1437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260924 02:58:03.542763 9237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:03.539181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40259 (local address 127.0.98.62:38311)
0924 02:58:03.539222 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:03.539223 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:03.539231 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:58:03.539253 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:03.539254 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:03.539261 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:03.539308 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:03.539503 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.539957 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.540528 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.540691 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.540733 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.542708 (+ 1975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40259: BlockingRecv error: recv got EOF from 127.0.98.2:40259 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:03.547025 9369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:03.542819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:58635)
0924 02:58:03.542913 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:03.542922 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:03.542961 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:03.543077 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:03.543078 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:03.543177 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:03.543201 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.543202 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.543908 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.543909 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.544594 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.544596 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.545014 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.545015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.545059 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.545429 (+ 370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:03.545434 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:03.546984 (+ 1550us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20260924 02:58:03.547128 9237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:03.542871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58635 (local address 127.0.98.62:38311)
0924 02:58:03.542912 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:03.542913 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:03.542952 (+ 39us) server_negotiation.cc:407] Connection header received
0924 02:58:03.543004 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:03.543005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:03.543013 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:03.543069 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:03.543255 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.543835 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.544669 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.544924 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.544987 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.547074 (+ 2087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:58635: BlockingRecv error: recv got EOF from 127.0.98.2:58635 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 02:58:03.547125 9367 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 02:58:03.621022 9370 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:03.621217 9370 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:03.621239 9370 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:03.622396 9370 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:03.622439 9370 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:03.622467 9370 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:58:03.623878 9370 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:45353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/krb5kdc/oryx_127.0.98.3.keytab
--principal=oryx/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:44431
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=39633
--tserver_master_addrs=127.0.98.62:38311
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9370
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:03.624099 9370 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:03.624308 9370 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:03.628908 9376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:03.628888 9375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:03.629045 9370 server_base.cc:1034] running on GCE node
W20260924 02:58:03.628893 9378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:03.629293 9370 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:03.629534 9370 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:03.630662 9370 hybrid_clock.cc:648] HybridClock initialized: now 1790218683630649 us; error 26 us; skew 500 ppm
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218683, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:58:03.632939 9370 init.cc:377] Logged in from keytab as oryx/127.0.98.3@KRBTEST.COM (short username oryx)
I20260924 02:58:03.633302 9370 webserver.cc:460] Webserver started at http://127.0.98.3:39633/ using document root <none> and password file <none>
I20260924 02:58:03.633496 9370 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:03.633545 9370 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:03.634760 9370 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:58:03.635257 9385 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:03.635403 9370 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:03.635478 9370 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "a651cf9b65384056acbd0c4cf6738231"
format_stamp: "Formatted at 2026-09-24 02:58:03 on dist-test-slave-sb6p"
server_key: "4cad209cece93eb3f984d70f32b45f1e"
server_key_iv: "b5e182212bcb492e6f3255ed451a1b26"
server_key_version: "encryptionkey@0"
I20260924 02:58:03.635788 9370 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:03.650645 9370 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:03.652594 9370 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:03.652726 9370 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:03.652951 9370 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:58:03.653206 9370 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:58:03.653234 9370 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:03.653270 9370 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:58:03.653282 9370 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:03.658630 9370 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44431
I20260924 02:58:03.658689 9498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44431 every 8 connection(s)
I20260924 02:58:03.659006 9370 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
W20260924 02:58:03.665537 9237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:03.659682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:56411 (local address 127.0.98.62:38311)
0924 02:58:03.659735 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:03.659737 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:03.659941 (+ 204us) server_negotiation.cc:407] Connection header received
0924 02:58:03.660385 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:03.660386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:03.660396 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:03.660452 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:03.661319 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.661805 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.662798 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.662984 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.663035 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.665491 (+ 2456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:56411: BlockingRecv error: recv got EOF from 127.0.98.3:56411 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 02:58:03.665550 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:03.659664 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:56411)
0924 02:58:03.659812 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:03.659848 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:03.660310 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:03.660512 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:03.660517 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:03.661052 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:03.661245 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.661250 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.661866 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.661866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.662732 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.662735 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.663043 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.663044 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.663136 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.663546 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:03.663554 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:03.665404 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
W20260924 02:58:03.665750 9499 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:38311 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.98.62:38311: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
I20260924 02:58:03.667092 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 9370
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:03.670140 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:03.665960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:35015)
0924 02:58:03.666001 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:03.666011 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:03.666046 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:03.666218 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:03.666218 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:03.666334 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:03.666372 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.666373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.666954 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.666955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.667566 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.667569 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.667957 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.667957 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.668002 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.668400 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:03.668421 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:03.670072 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":26}
W20260924 02:58:03.670171 9237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:03.666098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:35015 (local address 127.0.98.62:38311)
0924 02:58:03.666141 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:03.666143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:03.666148 (+ 5us) server_negotiation.cc:407] Connection header received
0924 02:58:03.666164 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:03.666165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:03.666172 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:03.666214 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:03.666419 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.666889 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.667629 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.667892 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.667955 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.670128 (+ 2173us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:35015: BlockingRecv error: recv got EOF from 127.0.98.3:35015 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:03 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:03.674377 9501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:03.670573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:44061)
0924 02:58:03.670602 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:03.670612 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:03.670646 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:03.670757 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:03.670757 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:03.670850 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:03.670873 (+ 23us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.670874 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.671492 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.671493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.672065 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:03.672067 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.672369 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.672370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:03.672424 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.672751 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:03.672754 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:03.674314 (+ 1560us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":14}
W20260924 02:58:03.674430 9237 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:03.670625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44061 (local address 127.0.98.62:38311)
0924 02:58:03.670662 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:03.670664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:03.670670 (+ 6us) server_negotiation.cc:407] Connection header received
0924 02:58:03.670687 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:03.670688 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:03.670695 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:03.670753 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:03.670931 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.671420 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.672136 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.672316 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.672366 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.674377 (+ 2011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44061: BlockingRecv error: recv got EOF from 127.0.98.3:44061 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 02:58:03.674499 9499 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 02:58:03.675017 9502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:03.667430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52636 (local address 127.0.98.62:38311)
0924 02:58:03.667530 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:03.667532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:03.667640 (+ 108us) server_negotiation.cc:407] Connection header received
0924 02:58:03.667691 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:03.667692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:03.667703 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:03.667747 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:03.668495 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.669135 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.670061 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:03.670389 (+ 328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:03.670470 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:03.671769 (+ 1299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:03.671773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:03.671774 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:58:03.671784 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:03.673392 (+ 1608us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:03.673734 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:03.673738 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:03.673738 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:03.673862 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:03.674027 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:03.674028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:03.674028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:03.674343 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:03.674479 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:03.674869 (+ 390us) server_negotiation.cc:299] Negotiation successful
0924 02:58:03.674959 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":57,"threads_started":1}
Sep 24 02:58:04 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:04 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:04.431977 9504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:04.427595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:36347)
0924 02:58:04.427711 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:04.427724 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:04.427779 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:04.427920 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:04.427922 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:04.428078 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:04.428121 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:04.428123 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:04.428761 (+ 638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:04.428762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:04.429311 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:04.429314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:04.429647 (+ 333us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:04.429648 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:04.429698 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:04.430108 (+ 410us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:04.430112 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:04.431886 (+ 1774us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
W20260924 02:58:04.432096 9505 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:04.427640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36347 (local address 127.0.98.62:38311)
0924 02:58:04.427751 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:04.427752 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:04.427764 (+ 12us) server_negotiation.cc:407] Connection header received
0924 02:58:04.427844 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:04.427845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:04.427856 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:04.427910 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:04.428196 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:04.428690 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:04.429397 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:04.429588 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:04.429639 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:04.432019 (+ 2380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:36347: BlockingRecv error: recv got EOF from 127.0.98.1:36347 (error 108)
Metrics: {"server-negotiator.queue_time_us":88,"thread_start_us":54,"threads_started":1}
Sep 24 02:58:04 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:04 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:04.551745 9506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:04.547460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:47403)
0924 02:58:04.547612 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:04.547623 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:04.547685 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:04.547819 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:04.547820 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:04.547974 (+ 154us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:04.548023 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:04.548024 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:04.548625 (+ 601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:04.548626 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:04.549171 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:04.549172 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:04.549477 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:04.549478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:04.549519 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:04.549919 (+ 400us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:04.549923 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:04.551670 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":64,"threads_started":1}
W20260924 02:58:04.551846 9505 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:04.547544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47403 (local address 127.0.98.62:38311)
0924 02:58:04.547593 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:04.547594 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:04.547682 (+ 88us) server_negotiation.cc:407] Connection header received
0924 02:58:04.547736 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:04.547736 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:04.547747 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:04.547784 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:04.548079 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:04.548569 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:04.549238 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:04.549418 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:04.549471 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:04.551781 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:47403: BlockingRecv error: recv got EOF from 127.0.98.2:47403 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 02:58:04 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:04 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:04.679167 9507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:04.674861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:39575)
0924 02:58:04.675005 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:04.675017 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:04.675081 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:04.675214 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:04.675215 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:04.675365 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:04.675413 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:04.675415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:04.676019 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:04.676020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:04.676560 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:04.676562 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:04.676864 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:04.676864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:04.676910 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:04.677356 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:04.677361 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:04.679077 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
W20260924 02:58:04.679230 9505 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:04.674911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39575 (local address 127.0.98.62:38311)
0924 02:58:04.674990 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:04.674992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:04.675066 (+ 74us) server_negotiation.cc:407] Connection header received
0924 02:58:04.675143 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:04.675144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:04.675159 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:04.675200 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:04.675468 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:04.675950 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:04.676621 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:04.676797 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:04.676839 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:04.679178 (+ 2339us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:39575: BlockingRecv error: recv got EOF from 127.0.98.3:39575 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 02:58:05 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:05 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:05.436709 9508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:05.432430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:51485)
0924 02:58:05.432558 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:05.432569 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:05.432624 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:05.432743 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:05.432744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:05.432892 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:05.432956 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:05.432958 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:05.433567 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:05.433567 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:05.434096 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:05.434098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:05.434387 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:05.434388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:05.434432 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:05.434895 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:05.434901 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:05.436653 (+ 1752us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":62,"threads_started":1}
W20260924 02:58:05.436856 9509 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:05.432527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51485 (local address 127.0.98.62:38311)
0924 02:58:05.432626 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:05.432629 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:05.432640 (+ 11us) server_negotiation.cc:407] Connection header received
0924 02:58:05.432666 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:05.432666 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:05.432676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:05.432731 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:05.433012 (+ 281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:05.433506 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:05.434153 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:05.434329 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:05.434382 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:05.436758 (+ 2376us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51485: BlockingRecv error: recv got EOF from 127.0.98.1:51485 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":44,"threads_started":1}
Sep 24 02:58:05 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:05 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:05.556352 9510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:05.552197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:41335)
0924 02:58:05.552319 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:05.552331 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:05.552399 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:05.552567 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:05.552568 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:05.552762 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:05.552809 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:05.552812 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:05.553475 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:05.553477 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:05.554005 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:05.554007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:05.554286 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:05.554287 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:05.554320 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:05.554734 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:05.554738 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:05.556297 (+ 1559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":57,"threads_started":1}
W20260924 02:58:05.556445 9509 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:05.552282 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41335 (local address 127.0.98.62:38311)
0924 02:58:05.552331 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:05.552333 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:05.552390 (+ 57us) server_negotiation.cc:407] Connection header received
0924 02:58:05.552470 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:05.552470 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:05.552485 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:05.552527 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:05.552871 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:05.553414 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:05.554065 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:05.554239 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:05.554281 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:05.556380 (+ 2099us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:41335: BlockingRecv error: recv got EOF from 127.0.98.2:41335 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 24 02:58:05 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:05 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:05.683816 9511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:05.679638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:36313)
0924 02:58:05.679782 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:05.679792 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:05.679853 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:05.679972 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:05.679973 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:05.680121 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:05.680169 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:05.680170 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:05.680755 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:05.680756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:05.681461 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:05.681465 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:05.681755 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:05.681755 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:05.681790 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:05.682215 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:05.682219 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:05.683758 (+ 1539us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":74,"threads_started":1}
W20260924 02:58:05.683902 9509 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:05.679756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36313 (local address 127.0.98.62:38311)
0924 02:58:05.679801 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:05.679803 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:05.679839 (+ 36us) server_negotiation.cc:407] Connection header received
0924 02:58:05.679894 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:05.679895 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:05.679907 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:05.679950 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:05.680219 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:05.680693 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:05.681519 (+ 826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:05.681701 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:05.681752 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:05.683843 (+ 2091us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:36313: BlockingRecv error: recv got EOF from 127.0.98.3:36313 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 24 02:58:06 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:06 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:06.445006 9512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:06.437197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:48517)
0924 02:58:06.437322 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:06.437332 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:06.437388 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:06.437547 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:06.437548 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:06.437703 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:06.437749 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:06.437751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:06.438367 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:06.438368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:06.438946 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:06.438948 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:06.439285 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:06.439286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:06.439336 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:06.443215 (+ 3879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:06.443220 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:06.444914 (+ 1694us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":67,"threads_started":1}
W20260924 02:58:06.445080 9513 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:06.437302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48517 (local address 127.0.98.62:38311)
0924 02:58:06.437426 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:06.437428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:06.437438 (+ 10us) server_negotiation.cc:407] Connection header received
0924 02:58:06.437464 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:06.437464 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:06.437473 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:06.437527 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:06.437807 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:06.438307 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:06.439029 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:06.439225 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:06.439281 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:06.445018 (+ 5737us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:48517: BlockingRecv error: recv got EOF from 127.0.98.1:48517 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
Sep 24 02:58:06 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:06 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:06.561228 9514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:06.556777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:43557)
0924 02:58:06.556914 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:06.556925 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:06.556990 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:06.557114 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:06.557115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:06.557295 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:06.557343 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:06.557345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:06.558023 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:06.558024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:06.558593 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:06.558595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:06.558902 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:06.558902 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:06.558950 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:06.559398 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:06.559403 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:06.561140 (+ 1737us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":80,"threads_started":1}
W20260924 02:58:06.561283 9513 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:06.556849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43557 (local address 127.0.98.62:38311)
0924 02:58:06.556907 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:06.556909 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:06.556993 (+ 84us) server_negotiation.cc:407] Connection header received
0924 02:58:06.557042 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:06.557043 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:06.557056 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:06.557103 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:06.557420 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:06.557929 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:06.558663 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:06.558844 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:06.558897 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:06.561234 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:43557: BlockingRecv error: recv got EOF from 127.0.98.2:43557 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
Sep 24 02:58:06 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:06 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:06.688704 9515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:06.684281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:50795)
0924 02:58:06.684427 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:06.684440 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:06.684513 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:06.684640 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:06.684641 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:06.684804 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:06.684848 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:06.684850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:06.685526 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:06.685527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:06.686147 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:06.686150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:06.686478 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:06.686479 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:06.686563 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:06.686984 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:06.686988 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:06.688617 (+ 1629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
W20260924 02:58:06.688759 9513 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:06.684376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50795 (local address 127.0.98.62:38311)
0924 02:58:06.684434 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:06.684436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:06.684493 (+ 57us) server_negotiation.cc:407] Connection header received
0924 02:58:06.684570 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:06.684571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:06.684585 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:06.684630 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:06.684921 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:06.685435 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:06.686223 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:06.686416 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:06.686475 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:06.688705 (+ 2230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:50795: BlockingRecv error: recv got EOF from 127.0.98.3:50795 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 24 02:58:07 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:07 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:07.449891 9516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:07.445448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:35515)
0924 02:58:07.445563 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:07.445573 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:07.445630 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:07.445792 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:07.445793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:07.445994 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:07.446042 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:07.446044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:07.446745 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:07.446747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:07.447274 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:07.447276 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:07.447591 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:07.447592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:07.447637 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:07.448029 (+ 392us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:07.448033 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:07.449831 (+ 1798us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
W20260924 02:58:07.450013 9517 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:07.445494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35515 (local address 127.0.98.62:38311)
0924 02:58:07.445600 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:07.445602 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:07.445614 (+ 12us) server_negotiation.cc:407] Connection header received
0924 02:58:07.445703 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:07.445704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:07.445717 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:07.445774 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:07.446106 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:07.446673 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:07.447341 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:07.447529 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:07.447572 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:07.449926 (+ 2354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35515: BlockingRecv error: recv got EOF from 127.0.98.1:35515 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":53,"threads_started":1}
Sep 24 02:58:07 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:07 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:07.565943 9518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:07.561677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:33885)
0924 02:58:07.561802 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:07.561813 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:07.561882 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:07.562026 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:07.562027 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:07.562192 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:07.562237 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:07.562239 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:07.562919 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:07.562920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:07.563438 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:07.563439 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:07.563755 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:07.563756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:07.563807 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:07.564211 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:07.564215 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:07.565866 (+ 1651us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
W20260924 02:58:07.566020 9517 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:07.561743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:33885 (local address 127.0.98.62:38311)
0924 02:58:07.561800 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:07.561803 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:07.561862 (+ 59us) server_negotiation.cc:407] Connection header received
0924 02:58:07.561942 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:07.561943 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:07.561954 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:07.562014 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:07.562304 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:07.562848 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:07.563515 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:07.563699 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:07.563750 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:07.565960 (+ 2210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:33885: BlockingRecv error: recv got EOF from 127.0.98.2:33885 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 02:58:07 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:07 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:07.693703 9519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:07.689181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:59505)
0924 02:58:07.689323 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:07.689335 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:07.689399 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:07.689528 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:07.689529 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:07.689689 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:07.689732 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:07.689734 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:07.690363 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:07.690364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:07.691008 (+ 644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:07.691010 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:07.691313 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:07.691314 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:07.691367 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:07.691808 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:07.691812 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:07.693625 (+ 1813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
W20260924 02:58:07.693768 9517 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:07.689264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59505 (local address 127.0.98.62:38311)
0924 02:58:07.689319 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:07.689321 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:07.689389 (+ 68us) server_negotiation.cc:407] Connection header received
0924 02:58:07.689459 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:07.689460 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:07.689474 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:07.689518 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:07.689797 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:07.690291 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:07.691080 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:07.691258 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:07.691308 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:07.693714 (+ 2406us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:59505: BlockingRecv error: recv got EOF from 127.0.98.3:59505 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 02:58:08 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:08 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:08.454494 9520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:08.450305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:33585)
0924 02:58:08.450432 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:08.450443 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:08.450498 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:08.450650 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:08.450650 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:08.450800 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:08.450847 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:08.450848 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:08.451485 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:08.451487 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:08.452057 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:08.452059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:08.452399 (+ 340us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:08.452400 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:08.452451 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:08.452837 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:08.452841 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:08.454434 (+ 1593us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
W20260924 02:58:08.454660 9521 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:08.450350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33585 (local address 127.0.98.62:38311)
0924 02:58:08.450461 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:08.450463 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:08.450489 (+ 26us) server_negotiation.cc:407] Connection header received
0924 02:58:08.450576 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:08.450577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:08.450587 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:08.450639 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:08.450897 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:08.451419 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:08.452128 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:08.452319 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:08.452401 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:08.454584 (+ 2183us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:33585: BlockingRecv error: recv got EOF from 127.0.98.1:33585 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":58,"threads_started":1}
Sep 24 02:58:08 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:08 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:08.570886 9522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:08.566421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:40787)
0924 02:58:08.566576 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:08.566589 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:08.566649 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:08.566780 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:08.566781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:08.566945 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:08.566991 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:08.567014 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:08.567675 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:08.567676 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:08.568365 (+ 689us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:08.568369 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:08.568684 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:08.568685 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:08.568736 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:08.569165 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:08.569171 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:08.570825 (+ 1654us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":72,"threads_started":1}
W20260924 02:58:08.570940 9521 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:08.566552 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40787 (local address 127.0.98.62:38311)
0924 02:58:08.566622 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:08.566624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:08.566633 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:58:08.566709 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:08.566710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:08.566721 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:08.566765 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:08.567090 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:08.567596 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:08.568435 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:08.568628 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:08.568680 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:08.570890 (+ 2210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:40787: BlockingRecv error: recv got EOF from 127.0.98.2:40787 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
Sep 24 02:58:08 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:08 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:08.698374 9523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:08.694185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:44619)
0924 02:58:08.694321 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:08.694332 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:08.694391 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:08.694511 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:08.694511 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:08.694700 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:08.694747 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:08.694749 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:08.695328 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:08.695328 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:08.695836 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:08.695837 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:08.696124 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:08.696124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:08.696157 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:08.696544 (+ 387us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:08.696548 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:08.698315 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":65,"threads_started":1}
W20260924 02:58:08.698457 9521 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:08.694251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44619 (local address 127.0.98.62:38311)
0924 02:58:08.694318 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:08.694320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:08.694382 (+ 62us) server_negotiation.cc:407] Connection header received
0924 02:58:08.694434 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:08.694434 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:08.694447 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:08.694494 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:08.694799 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:08.695276 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:08.695895 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:08.696072 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:08.696121 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:08.698399 (+ 2278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44619: BlockingRecv error: recv got EOF from 127.0.98.3:44619 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 24 02:58:09 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:09 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:09.459358 9524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:09.454942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:37463)
0924 02:58:09.455058 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:09.455068 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:09.455118 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:09.455265 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:09.455266 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:09.455414 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:09.455462 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:09.455464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:09.456077 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:09.456077 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:09.456633 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:09.456634 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:09.457033 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:09.457034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:09.457077 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:09.457552 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:09.457557 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:09.459271 (+ 1714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":68,"threads_started":1}
W20260924 02:58:09.459429 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:09.455046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:37463 (local address 127.0.98.62:38311)
0924 02:58:09.455154 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:09.455156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:09.455163 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:58:09.455185 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:09.455185 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:09.455195 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:09.455256 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:09.455518 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:09.456018 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:09.456745 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:09.456960 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:09.457036 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:09.459368 (+ 2332us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:37463: BlockingRecv error: recv got EOF from 127.0.98.1:37463 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":52,"threads_started":1}
Sep 24 02:58:09 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:09 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:09.575603 9526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:09.571344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:45681)
0924 02:58:09.571484 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:09.571495 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:09.571561 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:09.571712 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:09.571713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:09.571875 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:09.571927 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:09.571929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:09.572546 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:09.572547 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:09.573061 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:09.573062 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:09.573338 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:09.573339 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:09.573373 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:09.573787 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:09.573791 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:09.575534 (+ 1743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
W20260924 02:58:09.575662 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:09.571411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:45681 (local address 127.0.98.62:38311)
0924 02:58:09.571471 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:09.571473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:09.571555 (+ 82us) server_negotiation.cc:407] Connection header received
0924 02:58:09.571617 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:09.571617 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:09.571629 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:09.571666 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:09.572003 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:09.572487 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:09.573119 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:09.573292 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:09.573334 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:09.575606 (+ 2272us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:45681: BlockingRecv error: recv got EOF from 127.0.98.2:45681 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 24 02:58:09 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:09 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:09.703261 9527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:09.698860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:58135)
0924 02:58:09.699037 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:09.699048 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:09.699120 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:09.699250 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:09.699251 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:09.699421 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:09.699467 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:09.699469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:09.700111 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:09.700112 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:09.700641 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:09.700643 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:09.700948 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:09.700949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:09.701010 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:09.701406 (+ 396us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:09.701410 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:09.703174 (+ 1764us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":92,"threads_started":1}
W20260924 02:58:09.703321 9525 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:09.698956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58135 (local address 127.0.98.62:38311)
0924 02:58:09.699013 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:09.699016 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:09.699109 (+ 93us) server_negotiation.cc:407] Connection header received
0924 02:58:09.699175 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:09.699176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:09.699190 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:09.699236 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:09.699537 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:09.700037 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:09.700713 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:09.700891 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:09.700943 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:09.703271 (+ 2328us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58135: BlockingRecv error: recv got EOF from 127.0.98.3:58135 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
Sep 24 02:58:10 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:10 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:10.464322 9528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:10.459908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:35769)
0924 02:58:10.460046 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:10.460057 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:10.460114 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:10.460239 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:10.460239 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:10.460389 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:10.460454 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:10.460455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:10.461075 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:10.461075 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:10.461718 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:10.461721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:10.462134 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:10.462134 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:10.462187 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:10.462658 (+ 471us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:10.462662 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:10.464257 (+ 1595us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":62,"threads_started":1}
W20260924 02:58:10.464414 9529 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:10.459993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35769 (local address 127.0.98.62:38311)
0924 02:58:10.460098 (+ 105us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:10.460100 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:10.460108 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:58:10.460159 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:10.460160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:10.460170 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:10.460226 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:10.460532 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:10.461018 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:10.461782 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:10.462073 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:10.462123 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:10.464344 (+ 2221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:35769: BlockingRecv error: recv got EOF from 127.0.98.1:35769 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":51,"threads_started":1}
Sep 24 02:58:10 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:10 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:10.580228 9530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:10.576048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:44469)
0924 02:58:10.576172 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:10.576183 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:10.576242 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:10.576354 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:10.576355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:10.576503 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:10.576572 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:10.576574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:10.577179 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:10.577180 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:10.577733 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:10.577735 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:10.578046 (+ 311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:10.578047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:10.578083 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:10.578511 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:10.578554 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:10.580152 (+ 1598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
W20260924 02:58:10.580302 9529 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:10.576114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44469 (local address 127.0.98.62:38311)
0924 02:58:10.576177 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:10.576180 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:10.576235 (+ 55us) server_negotiation.cc:407] Connection header received
0924 02:58:10.576294 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:10.576295 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:10.576305 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:10.576345 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:10.576623 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:10.577118 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:10.577802 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:10.577995 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:10.578041 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:10.580241 (+ 2200us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:44469: BlockingRecv error: recv got EOF from 127.0.98.2:44469 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 24 02:58:10 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:10 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:10.707852 9531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:10.703711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:45701)
0924 02:58:10.703853 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:10.703864 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:10.703917 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:10.704056 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:10.704057 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:10.704202 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:10.704251 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:10.704253 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:10.704855 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:10.704856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:10.705406 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:10.705407 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:10.705709 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:10.705710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:10.705745 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:10.706167 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:10.706172 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:10.707795 (+ 1623us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":70,"threads_started":1}
W20260924 02:58:10.707937 9529 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:10.703780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:45701 (local address 127.0.98.62:38311)
0924 02:58:10.703827 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:10.703829 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:10.703910 (+ 81us) server_negotiation.cc:407] Connection header received
0924 02:58:10.703962 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:10.703962 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:10.703976 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:10.704035 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:10.704301 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:10.704796 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:10.705465 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:10.705658 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:10.705702 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:10.707873 (+ 2171us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:45701: BlockingRecv error: recv got EOF from 127.0.98.3:45701 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
Sep 24 02:58:11 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:11 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:11.473559 9532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:11.464779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:45529)
0924 02:58:11.464917 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:11.464927 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:11.464999 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:11.465143 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:11.465143 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:11.465298 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:11.465347 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:11.465349 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:11.465938 (+ 589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:11.465939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:11.466498 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:11.466500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:11.466922 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:11.466924 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:11.466982 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:11.471784 (+ 4802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:11.471789 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:11.473492 (+ 1703us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":55,"threads_started":1}
W20260924 02:58:11.473656 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:11.464919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:45529 (local address 127.0.98.62:38311)
0924 02:58:11.465033 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:11.465035 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:11.465043 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:58:11.465065 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:11.465066 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:11.465075 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:11.465130 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:11.465396 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:11.465884 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:11.466595 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:11.466793 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:11.466844 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:11.473583 (+ 6739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:45529: BlockingRecv error: recv got EOF from 127.0.98.1:45529 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":55,"threads_started":1}
Sep 24 02:58:11 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:11 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:11.585091 9534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:11.580653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:46641)
0924 02:58:11.580762 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:11.580772 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:11.580825 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:11.580938 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:11.580939 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:11.581106 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:11.581167 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:11.581169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:11.581958 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:11.581958 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:11.582544 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:11.582546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:11.582858 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:11.582859 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:11.582897 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:11.583316 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:11.583320 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:11.585015 (+ 1695us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":62,"threads_started":1}
W20260924 02:58:11.585140 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:11.580726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46641 (local address 127.0.98.62:38311)
0924 02:58:11.580769 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:11.580772 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:11.580811 (+ 39us) server_negotiation.cc:407] Connection header received
0924 02:58:11.580865 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:11.580866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:11.580876 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:11.580934 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:11.581221 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:11.581885 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:11.582620 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:11.582802 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:11.582854 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:11.585094 (+ 2240us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:46641: BlockingRecv error: recv got EOF from 127.0.98.2:46641 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
Sep 24 02:58:11 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:11 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:11.712378 9535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:11.708267 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:37989)
0924 02:58:11.708431 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:11.708441 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:11.708493 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:11.708626 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:11.708627 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:11.708775 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:11.708820 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:11.708822 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:11.709425 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:11.709426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:11.709965 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:11.709967 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:11.710249 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:11.710250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:11.710285 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:11.710720 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:11.710725 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:11.712302 (+ 1577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
W20260924 02:58:11.712442 9533 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:11.708321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37989 (local address 127.0.98.62:38311)
0924 02:58:11.708399 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:11.708400 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:11.708479 (+ 79us) server_negotiation.cc:407] Connection header received
0924 02:58:11.708553 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:11.708553 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:11.708565 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:11.708610 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:11.708867 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:11.709355 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:11.710022 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:11.710198 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:11.710245 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:11.712390 (+ 2145us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:37989: BlockingRecv error: recv got EOF from 127.0.98.3:37989 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 24 02:58:12 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:12 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:12.482863 9536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:12.474022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:57203)
0924 02:58:12.474142 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:12.474152 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:12.474199 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:12.474326 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:12.474327 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:12.474477 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:12.474554 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:12.474556 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:12.475195 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:12.475195 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:12.475766 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:12.475768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:12.476107 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:12.476108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:12.476143 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:12.480959 (+ 4816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:12.480965 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:12.482793 (+ 1828us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":60,"threads_started":1}
W20260924 02:58:12.482963 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:12.474084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57203 (local address 127.0.98.62:38311)
0924 02:58:12.474194 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:12.474196 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:12.474204 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:58:12.474243 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:12.474244 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:12.474255 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:12.474316 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:12.474616 (+ 300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:12.475131 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:12.475849 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:12.476049 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:12.476102 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:12.482887 (+ 6785us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57203: BlockingRecv error: recv got EOF from 127.0.98.1:57203 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":59,"threads_started":1}
Sep 24 02:58:12 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:12 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:12.589649 9538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:12.585545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:48123)
0924 02:58:12.585650 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:12.585660 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:12.585706 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:12.585833 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:12.585833 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:12.585984 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:12.586030 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:12.586032 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:12.586673 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:12.586673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:12.587182 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:12.587184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:12.587462 (+ 278us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:12.587462 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:12.587518 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:12.587909 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:12.587913 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:12.589560 (+ 1647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":60,"threads_started":1}
W20260924 02:58:12.589740 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:12.585612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:48123 (local address 127.0.98.62:38311)
0924 02:58:12.585687 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:12.585689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:12.585700 (+ 11us) server_negotiation.cc:407] Connection header received
0924 02:58:12.585757 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:12.585758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:12.585769 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:12.585821 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:12.586079 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:12.586620 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:12.587237 (+ 617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:12.587416 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:12.587458 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:12.589683 (+ 2225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:48123: BlockingRecv error: recv got EOF from 127.0.98.2:48123 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Sep 24 02:58:12 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:12 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:12.717159 9539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:12.712880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:44851)
0924 02:58:12.712995 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:12.713007 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:12.713071 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:12.713218 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:12.713219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:12.713402 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:12.713451 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:12.713453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:12.714084 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:12.714085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:12.714656 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:12.714659 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:12.715000 (+ 341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:12.715001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:12.715055 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:12.715432 (+ 377us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:12.715436 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:12.717102 (+ 1666us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
W20260924 02:58:12.717243 9537 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:12.712941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44851 (local address 127.0.98.62:38311)
0924 02:58:12.712999 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:12.713002 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:12.713057 (+ 55us) server_negotiation.cc:407] Connection header received
0924 02:58:12.713143 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:12.713144 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:12.713157 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:12.713201 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:12.713524 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:12.714012 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:12.714721 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:12.714938 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:12.714993 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:12.717183 (+ 2190us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44851: BlockingRecv error: recv got EOF from 127.0.98.3:44851 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 02:58:13 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:13 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:13.487428 9540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:13.483307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:57115)
0924 02:58:13.483477 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:13.483489 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:13.483540 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:13.483657 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:13.483658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:13.483808 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:13.483857 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:13.483859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:13.484478 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:13.484478 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:13.484998 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:13.485000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:13.485305 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:13.485305 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:13.485340 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:13.485752 (+ 412us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:13.485756 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:13.487370 (+ 1614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":113,"threads_started":1}
W20260924 02:58:13.487524 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:13.483352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57115 (local address 127.0.98.62:38311)
0924 02:58:13.483466 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:13.483468 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:13.483528 (+ 60us) server_negotiation.cc:407] Connection header received
0924 02:58:13.483585 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:13.483585 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:13.483595 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:13.483649 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:13.483908 (+ 259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:13.484406 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:13.485065 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:13.485252 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:13.485299 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:13.487455 (+ 2156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:57115: BlockingRecv error: recv got EOF from 127.0.98.1:57115 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
Sep 24 02:58:13 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:13 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:13.594393 9542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:13.590089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:37209)
0924 02:58:13.590205 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:13.590217 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:13.590278 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:13.590401 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:13.590402 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:13.590624 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:13.590694 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:13.590696 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:13.591379 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:13.591380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:13.591966 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:13.591968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:13.592293 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:13.592294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:13.592343 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:13.592762 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:13.592767 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:13.594332 (+ 1565us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":62,"threads_started":1}
W20260924 02:58:13.594471 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:13.590144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37209 (local address 127.0.98.62:38311)
0924 02:58:13.590200 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:13.590202 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:13.590291 (+ 89us) server_negotiation.cc:407] Connection header received
0924 02:58:13.590317 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:13.590318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:13.590331 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:13.590388 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:13.590767 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:13.591311 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:13.592040 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:13.592239 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:13.592292 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:13.594416 (+ 2124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:37209: BlockingRecv error: recv got EOF from 127.0.98.2:37209 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 24 02:58:13 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:13 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:13.722069 9543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:13.717589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:43641)
0924 02:58:13.717735 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:13.717745 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:13.717795 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:13.717912 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:13.717913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:13.718091 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:13.718134 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:13.718135 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:13.718930 (+ 795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:13.718932 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:13.719528 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:13.719531 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:13.719868 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:13.719870 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:13.719924 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:13.720368 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:13.720372 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:13.721992 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":68,"threads_started":1}
W20260924 02:58:13.722182 9541 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:13.717717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43641 (local address 127.0.98.62:38311)
0924 02:58:13.717772 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:13.717773 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:13.717783 (+ 10us) server_negotiation.cc:407] Connection header received
0924 02:58:13.717846 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:13.717846 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:13.717857 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:13.717901 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:13.718222 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:13.718842 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:13.719614 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:13.719807 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:13.719860 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:13.722082 (+ 2222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:43641: BlockingRecv error: recv got EOF from 127.0.98.3:43641 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 02:58:14 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:14 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:14.492337 9544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:14.487895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:46551)
0924 02:58:14.488017 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:14.488028 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:14.488093 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:14.488218 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:14.488219 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:14.488372 (+ 153us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:14.488421 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:14.488422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:14.489092 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:14.489092 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:14.489634 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:14.489635 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:14.489947 (+ 312us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:14.489948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:14.489987 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:14.490441 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:14.490446 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:14.492247 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":69,"threads_started":1}
W20260924 02:58:14.492405 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:14.487968 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46551 (local address 127.0.98.62:38311)
0924 02:58:14.488086 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:14.488088 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:14.488096 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:58:14.488138 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:14.488138 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:14.488147 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:14.488204 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:14.488513 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:14.489027 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:14.489707 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:14.489894 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:14.489944 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:14.492344 (+ 2400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:46551: BlockingRecv error: recv got EOF from 127.0.98.1:46551 (error 108)
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":63,"threads_started":1}
Sep 24 02:58:14 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:14 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:14.599282 9546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:14.594890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:54469)
0924 02:58:14.595007 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:14.595041 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:14.595111 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:14.595255 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:14.595256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:14.595425 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:14.595467 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:14.595469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:14.596121 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:14.596122 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:14.596626 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:14.596628 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:14.596942 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:14.596943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:14.596995 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:14.597436 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:14.597440 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:14.599222 (+ 1782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":56,"threads_started":1}
W20260924 02:58:14.599417 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:14.594997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54469 (local address 127.0.98.62:38311)
0924 02:58:14.595080 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:14.595081 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:14.595091 (+ 10us) server_negotiation.cc:407] Connection header received
0924 02:58:14.595173 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:14.595174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:14.595186 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:14.595248 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:14.595565 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:14.596051 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:14.596698 (+ 647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:14.596885 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:14.596937 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:14.599350 (+ 2413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54469: BlockingRecv error: recv got EOF from 127.0.98.2:54469 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Sep 24 02:58:14 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:14 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:14.726588 9547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:14.722480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:40495)
0924 02:58:14.722657 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:14.722668 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:14.722723 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:14.722837 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:14.722838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:14.722989 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:14.723059 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:14.723061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:14.723712 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:14.723713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:14.724238 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:14.724240 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:14.724526 (+ 286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:14.724526 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:14.724561 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:14.724992 (+ 431us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:14.724997 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:14.726508 (+ 1511us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":133,"threads_started":1}
W20260924 02:58:14.726693 9545 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:14.722560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40495 (local address 127.0.98.62:38311)
0924 02:58:14.722615 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:14.722617 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:14.722721 (+ 104us) server_negotiation.cc:407] Connection header received
0924 02:58:14.722764 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:14.722765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:14.722777 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:14.722830 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:14.723148 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:14.723650 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:14.724298 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:14.724474 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:14.724518 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:14.726624 (+ 2106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:40495: BlockingRecv error: recv got EOF from 127.0.98.3:40495 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 24 02:58:15 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:15 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:15.497126 9548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:15.492801 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:55897)
0924 02:58:15.492921 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:15.492932 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:15.492980 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:15.493146 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:15.493148 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:15.493308 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:15.493355 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:15.493356 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:15.493989 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:15.493990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:15.494507 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:15.494509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:15.494844 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:15.494844 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:15.494900 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:15.495322 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:15.495327 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:15.497060 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
W20260924 02:58:15.497205 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:15.492873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55897 (local address 127.0.98.62:38311)
0924 02:58:15.492991 (+ 118us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:15.492993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:15.493020 (+ 27us) server_negotiation.cc:407] Connection header received
0924 02:58:15.493043 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:15.493043 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:15.493053 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:15.493103 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:15.493415 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:15.493923 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:15.494613 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:15.494796 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:15.494839 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:15.497145 (+ 2306us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:55897: BlockingRecv error: recv got EOF from 127.0.98.1:55897 (error 108)
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":50,"threads_started":1}
Sep 24 02:58:15 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:15 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:15.604184 9550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:15.599733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:53559)
0924 02:58:15.599871 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:15.599885 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:15.599967 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:15.600119 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:15.600120 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:15.600288 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:15.600334 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:15.600336 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:15.600958 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:15.600959 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:15.601518 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:15.601520 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:15.601844 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:15.601845 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:15.601897 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:15.602286 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:15.602289 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:15.604127 (+ 1838us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
W20260924 02:58:15.604270 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:15.599836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53559 (local address 127.0.98.62:38311)
0924 02:58:15.599907 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:15.599911 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:15.599945 (+ 34us) server_negotiation.cc:407] Connection header received
0924 02:58:15.600047 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:15.600048 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:15.600058 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:15.600104 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:15.600406 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:15.600886 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:15.601599 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:15.601785 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:15.601838 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:15.604210 (+ 2372us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:53559: BlockingRecv error: recv got EOF from 127.0.98.2:53559 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 24 02:58:15 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:15 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:15.731303 9551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:15.727067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:58299)
0924 02:58:15.727214 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:15.727225 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:15.727289 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:15.727405 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:15.727406 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:15.727574 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:15.727618 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:15.727619 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:15.728253 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:15.728254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:15.728773 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:15.728775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:15.729091 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:15.729093 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:15.729146 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:15.729560 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:15.729565 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:15.731218 (+ 1653us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":86,"threads_started":1}
W20260924 02:58:15.731362 9549 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:15.727183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:58299 (local address 127.0.98.62:38311)
0924 02:58:15.727229 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:15.727231 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:15.727288 (+ 57us) server_negotiation.cc:407] Connection header received
0924 02:58:15.727340 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:15.727341 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:15.727355 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:15.727401 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:15.727690 (+ 289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:15.728181 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:15.728849 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:15.729032 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:15.729085 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:15.731310 (+ 2225us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:58299: BlockingRecv error: recv got EOF from 127.0.98.3:58299 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 24 02:58:16 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:16 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:16.505471 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:16.497572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:52131)
0924 02:58:16.497682 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:16.497694 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:16.497763 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:16.497888 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:16.497888 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:16.498044 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:16.498092 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:16.498093 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:16.498747 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:16.498748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:16.499288 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:16.499290 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:16.499613 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:16.499614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:16.499659 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:16.503576 (+ 3917us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:16.503581 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:16.505400 (+ 1819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":53,"threads_started":1}
W20260924 02:58:16.505586 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:16.497651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52131 (local address 127.0.98.62:38311)
0924 02:58:16.497782 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:16.497784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:16.497791 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:58:16.497810 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:16.497811 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:16.497821 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:16.497875 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:16.498154 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:16.498671 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:16.499358 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:16.499553 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:16.499600 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:16.505509 (+ 5909us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:52131: BlockingRecv error: recv got EOF from 127.0.98.1:52131 (error 108)
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
Sep 24 02:58:16 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:16 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:16.608816 9554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:16.604566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:50711)
0924 02:58:16.604690 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:16.604700 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:16.604746 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:16.604844 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:16.604845 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:16.604993 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:16.605046 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:16.605048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:16.605696 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:16.605696 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:16.606222 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:16.606223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:16.606584 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:16.606585 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:16.606619 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:16.607055 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:16.607060 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:16.608729 (+ 1669us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":55,"threads_started":1}
W20260924 02:58:16.608853 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:16.604695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50711 (local address 127.0.98.62:38311)
0924 02:58:16.604746 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:16.604748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:16.604756 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:58:16.604781 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:16.604781 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:16.604790 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:16.604835 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:16.605134 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:16.605620 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:16.606297 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:16.606502 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:16.606588 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:16.608803 (+ 2215us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:50711: BlockingRecv error: recv got EOF from 127.0.98.2:50711 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
Sep 24 02:58:16 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:16 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:16.735808 9555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:16.731757 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:53843)
0924 02:58:16.731869 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:16.731880 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:16.731945 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:16.732070 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:16.732071 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:16.732220 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:16.732266 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:16.732268 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:16.732890 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:16.732890 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:16.733405 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:16.733407 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:16.733721 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:16.733722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:16.733758 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:16.734180 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:16.734186 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:16.735753 (+ 1567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":60,"threads_started":1}
W20260924 02:58:16.735903 9553 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:16.731795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53843 (local address 127.0.98.62:38311)
0924 02:58:16.731843 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:16.731845 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:16.731935 (+ 90us) server_negotiation.cc:407] Connection header received
0924 02:58:16.732004 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:16.732005 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:16.732017 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:16.732060 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:16.732312 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:16.732834 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:16.733464 (+ 630us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:16.733669 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:16.733718 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:16.735837 (+ 2119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:53843: BlockingRecv error: recv got EOF from 127.0.98.3:53843 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 02:58:17 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:17 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:17.512320 9557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:17.507744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:51073)
0924 02:58:17.507882 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:17.507893 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:17.507954 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:17.508110 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:17.508112 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:17.508319 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:17.508377 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:17.508379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:17.509081 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:17.509082 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:17.509718 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:17.509720 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:17.510075 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:17.510076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:17.510113 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:17.510498 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:17.510502 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:17.512250 (+ 1748us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":61,"threads_started":1}
W20260924 02:58:17.512405 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:17.507711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51073 (local address 127.0.98.62:38311)
0924 02:58:17.507913 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:17.507915 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:17.507989 (+ 74us) server_negotiation.cc:407] Connection header received
0924 02:58:17.508013 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:17.508014 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:17.508027 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:17.508133 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:17.508457 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:17.509006 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:17.509798 (+ 792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:17.510008 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:17.510067 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:17.512336 (+ 2269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:51073: BlockingRecv error: recv got EOF from 127.0.98.1:51073 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
Sep 24 02:58:17 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:17 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:17.613584 9558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:17.609251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:60615)
0924 02:58:17.609381 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:17.609393 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:17.609460 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:17.609596 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:17.609596 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:17.609764 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:17.609825 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:17.609827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:17.610423 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:17.610424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:17.611004 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:17.611007 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:17.611334 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:17.611335 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:17.611387 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:17.611841 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:17.611846 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:17.613526 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":55,"threads_started":1}
W20260924 02:58:17.613686 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:17.609343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:60615 (local address 127.0.98.62:38311)
0924 02:58:17.609387 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:17.609389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:17.609445 (+ 56us) server_negotiation.cc:407] Connection header received
0924 02:58:17.609507 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:17.609507 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:17.609520 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:17.609584 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:17.609871 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:17.610351 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:17.611072 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:17.611259 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:17.611308 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:17.613629 (+ 2321us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:60615: BlockingRecv error: recv got EOF from 127.0.98.2:60615 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 24 02:58:17 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:17 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.3@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:17.740464 9559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:17.736244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.3:44643)
0924 02:58:17.736367 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:17.736380 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:17.736441 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:17.736595 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:17.736596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:17.736755 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:17.736799 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:17.736801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:17.737427 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:17.737428 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:17.737959 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:17.737961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:17.738292 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:17.738293 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:17.738345 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:17.738794 (+ 449us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:17.738800 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:17.740388 (+ 1588us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
W20260924 02:58:17.740528 9556 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:17.736344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:44643 (local address 127.0.98.62:38311)
0924 02:58:17.736392 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:17.736394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:17.736423 (+ 29us) server_negotiation.cc:407] Connection header received
0924 02:58:17.736496 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:17.736497 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:17.736510 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:17.736556 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:17.736866 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:17.737353 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:17.738032 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:17.738231 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:17.738283 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:17.740476 (+ 2193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.3:44643: BlockingRecv error: recv got EOF from 127.0.98.3:44643 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 02:58:18 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:18 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.1@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:18.517323 9560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:18.512773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.1:40003)
0924 02:58:18.512887 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:18.512896 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:18.512943 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:18.513103 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:18.513103 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:18.513246 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:18.513299 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:18.513301 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:18.514033 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:18.514034 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:18.514693 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:18.514695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:18.515141 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:18.515142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:18.515185 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:18.515582 (+ 397us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:18.515586 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:18.517266 (+ 1680us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
W20260924 02:58:18.517428 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:18.512870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40003 (local address 127.0.98.62:38311)
0924 02:58:18.512974 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:18.512976 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:18.512986 (+ 10us) server_negotiation.cc:407] Connection header received
0924 02:58:18.513017 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:18.513018 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:18.513029 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:18.513094 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:18.513391 (+ 297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:18.513953 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:18.514773 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:18.515079 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:18.515133 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:18.517360 (+ 2227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.1:40003: BlockingRecv error: recv got EOF from 127.0.98.1:40003 (error 108)
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":56,"threads_started":1}
Sep 24 02:58:18 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
Sep 24 02:58:18 dist-test-slave-sb6p krb5kdc[8503](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.98.2@KRBTEST.COM for oryx/127.0.98.62@KRBTEST.COM, Server not found in Kerberos database
I20260924 02:58:18.618373 9562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:18.614076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:38311 (local address 127.0.98.2:54823)
0924 02:58:18.614203 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:18.614214 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:18.614269 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:18.614447 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:18.614448 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:18.614630 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:18.614676 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:18.614678 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:18.615382 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:18.615383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:18.615945 (+ 562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:18.615947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:18.616254 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:18.616255 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:18.616293 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:18.616729 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:18.616733 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:18.618317 (+ 1584us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.98.62:38311: Server oryx/127.0.98.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
W20260924 02:58:18.618462 9561 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:18.614163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54823 (local address 127.0.98.62:38311)
0924 02:58:18.614228 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:18.614230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:18.614259 (+ 29us) server_negotiation.cc:407] Connection header received
0924 02:58:18.614313 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:18.614313 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:18.614323 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:18.614376 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:18.614775 (+ 399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:18.615262 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:18.616012 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:18.616200 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:18.616246 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:18.618405 (+ 2159us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.98.2:54823: BlockingRecv error: recv got EOF from 127.0.98.2:54823 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 02:58:18.674593 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 9086
I20260924 02:58:18.678622 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 9238
I20260924 02:58:18.681998 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 9370
I20260924 02:58:18.685719 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 9016
2026-09-24T02:58:18Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17272 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-24T02:58:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:58:18Z Disabled control of system clock
I20260924 02:58:18.710963 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36925
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:35129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36925
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260924 02:58:18.787704 9566 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:18.787894 9566 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:18.787912 9566 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:18.789124 9566 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:58:18.789162 9566 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:18.789175 9566 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:18.789187 9566 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:58:18.789211 9566 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:58:18.790509 9566 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36925
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9566
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:18.790750 9566 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:18.791015 9566 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:18.795807 9571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:18.795881 9574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:18.795863 9566 server_base.cc:1034] running on GCE node
W20260924 02:58:18.795882 9572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:18.796278 9566 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:18.796537 9566 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:18.797700 9566 hybrid_clock.cc:648] HybridClock initialized: now 1790218698797686 us; error 28 us; skew 500 ppm
I20260924 02:58:18.798940 9566 webserver.cc:460] Webserver started at http://127.0.98.62:35297/ using document root <none> and password file <none>
I20260924 02:58:18.799141 9566 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:18.799185 9566 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:18.799302 9566 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:58:18.800323 9566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "93a7393eb2d4425e8670fee93be9ee88"
format_stamp: "Formatted at 2026-09-24 02:58:18 on dist-test-slave-sb6p"
server_key: "ce7077496a2cfb3f1f5fe1feb1f34022"
server_key_iv: "6927e8c1d3676575703b5435926af7ff"
server_key_version: "encryptionkey@0"
I20260924 02:58:18.800638 9566 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "93a7393eb2d4425e8670fee93be9ee88"
format_stamp: "Formatted at 2026-09-24 02:58:18 on dist-test-slave-sb6p"
server_key: "ce7077496a2cfb3f1f5fe1feb1f34022"
server_key_iv: "6927e8c1d3676575703b5435926af7ff"
server_key_version: "encryptionkey@0"
I20260924 02:58:18.801875 9566 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:58:18.802570 9580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:18.802712 9566 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:58:18.802791 9566 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "93a7393eb2d4425e8670fee93be9ee88"
format_stamp: "Formatted at 2026-09-24 02:58:18 on dist-test-slave-sb6p"
server_key: "ce7077496a2cfb3f1f5fe1feb1f34022"
server_key_iv: "6927e8c1d3676575703b5435926af7ff"
server_key_version: "encryptionkey@0"
I20260924 02:58:18.802852 9566 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:18.814245 9566 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:18.814483 9566 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:18.814632 9566 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:18.817945 9566 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36925
I20260924 02:58:18.817981 9632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36925 every 8 connection(s)
I20260924 02:58:18.818266 9566 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:58:18.818897 9633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:58:18.821350 9633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88: Bootstrap starting.
I20260924 02:58:18.821887 9633 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88: Neither blocks nor log segments found. Creating new log.
I20260924 02:58:18.822129 9633 log.cc:824] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88: Log is configured to *not* fsync() on all Append() calls
I20260924 02:58:18.822788 9633 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88: No bootstrap required, opened a new log
I20260924 02:58:18.824259 9633 raft_consensus.cc:348] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a7393eb2d4425e8670fee93be9ee88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36925 } }
I20260924 02:58:18.824427 9633 raft_consensus.cc:374] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:58:18.824470 9633 raft_consensus.cc:729] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93a7393eb2d4425e8670fee93be9ee88, State: Initialized, Role: FOLLOWER
I20260924 02:58:18.824476 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 9566
I20260924 02:58:18.824548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:58:18.824625 9633 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a7393eb2d4425e8670fee93be9ee88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36925 } }
I20260924 02:58:18.824699 9633 raft_consensus.cc:388] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:58:18.824734 9633 raft_consensus.cc:482] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:58:18.824740 392 external_mini_cluster.cc:1428] Setting key e45a5d634006d1153575cbd49bd96a08
I20260924 02:58:18.824779 9633 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:58:18.825351 9633 raft_consensus.cc:504] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a7393eb2d4425e8670fee93be9ee88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36925 } }
I20260924 02:58:18.825433 9633 leader_election.cc:302] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 93a7393eb2d4425e8670fee93be9ee88; no voters:
I20260924 02:58:18.825574 9633 leader_election.cc:288] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:58:18.825611 9638 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:58:18.825739 9638 raft_consensus.cc:686] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [term 1 LEADER]: Becoming Leader. State: Replica: 93a7393eb2d4425e8670fee93be9ee88, State: Running, Role: LEADER
I20260924 02:58:18.825819 9633 sys_catalog.cc:564] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:58:18.825878 9638 consensus_queue.cc:237] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a7393eb2d4425e8670fee93be9ee88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36925 } }
I20260924 02:58:18.826303 9640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "93a7393eb2d4425e8670fee93be9ee88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a7393eb2d4425e8670fee93be9ee88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36925 } } }
I20260924 02:58:18.826398 9640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [sys.catalog]: This master's current role is: LEADER
I20260924 02:58:18.826318 9638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 93a7393eb2d4425e8670fee93be9ee88. Latest consensus state: current_term: 1 leader_uuid: "93a7393eb2d4425e8670fee93be9ee88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "93a7393eb2d4425e8670fee93be9ee88" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36925 } } }
I20260924 02:58:18.826732 9638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88 [sys.catalog]: This master's current role is: LEADER
I20260924 02:58:18.826915 9647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:58:18.827471 9647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:58:18.828789 9647 catalog_manager.cc:1347] Generated new cluster ID: 51f37677112347faaaf449c63d40d729
I20260924 02:58:18.828838 9647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:58:18.830935 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:18.825180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43004 (local address 127.0.98.62:36925)
0924 02:58:18.825372 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:18.825381 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:18.825409 (+ 28us) server_negotiation.cc:407] Connection header received
0924 02:58:18.825523 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:18.825530 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:18.825587 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:18.825711 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:18.826012 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:18.827760 (+ 1748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:18.828839 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:18.829443 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:18.829555 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:18.830224 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:18.830227 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:18.830228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:58:18.830437 (+ 209us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:18.830478 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:58:18.830492 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:18.830597 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:18.830658 (+ 61us) server_negotiation.cc:299] Negotiation successful
0924 02:58:18.830736 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260924 02:58:18.843174 9647 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:58:18.843673 9647 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:58:18.856650 9647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 93a7393eb2d4425e8670fee93be9ee88: Generated new TSK 0
I20260924 02:58:18.856818 9647 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 02:58:18.861188 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36925
--builtin_ntp_servers=127.0.98.20:35129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 02:58:18.938728 9657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:18.938923 9657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:18.938941 9657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:18.940044 9657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:18.940088 9657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:18.940111 9657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:58:18.941375 9657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9657
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:18.941579 9657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:18.941845 9657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:18.946475 9663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:18.946501 9662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:18.946501 9665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:18.946687 9657 server_base.cc:1034] running on GCE node
I20260924 02:58:18.946928 9657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:18.947162 9657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:18.948307 9657 hybrid_clock.cc:648] HybridClock initialized: now 1790218698948293 us; error 30 us; skew 500 ppm
I20260924 02:58:18.949411 9657 webserver.cc:460] Webserver started at http://127.0.98.1:39801/ using document root <none> and password file <none>
I20260924 02:58:18.949586 9657 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:18.949623 9657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:18.949708 9657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:58:18.950704 9657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "8c3431bd09c145948bf896c053868675"
format_stamp: "Formatted at 2026-09-24 02:58:18 on dist-test-slave-sb6p"
server_key: "9b44f60b3383a7f97c6ef014f51f708b"
server_key_iv: "d93f9e0835a6554b9cabd80bc4d3156c"
server_key_version: "encryptionkey@0"
I20260924 02:58:18.950980 9657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "8c3431bd09c145948bf896c053868675"
format_stamp: "Formatted at 2026-09-24 02:58:18 on dist-test-slave-sb6p"
server_key: "9b44f60b3383a7f97c6ef014f51f708b"
server_key_iv: "d93f9e0835a6554b9cabd80bc4d3156c"
server_key_version: "encryptionkey@0"
I20260924 02:58:18.952078 9657 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 02:58:18.952734 9671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:18.952893 9657 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:18.952975 9657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "8c3431bd09c145948bf896c053868675"
format_stamp: "Formatted at 2026-09-24 02:58:18 on dist-test-slave-sb6p"
server_key: "9b44f60b3383a7f97c6ef014f51f708b"
server_key_iv: "d93f9e0835a6554b9cabd80bc4d3156c"
server_key_version: "encryptionkey@0"
I20260924 02:58:18.953027 9657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:18.973646 9657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:18.973912 9657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:18.974009 9657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:18.974186 9657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:58:18.974444 9657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:58:18.974472 9657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:18.974496 9657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:58:18.974550 9657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:18.979740 9657 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44883
I20260924 02:58:18.979808 9784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44883 every 8 connection(s)
I20260924 02:58:18.980073 9657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:58:18.984584 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:18.980670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:40861 (local address 127.0.98.62:36925)
0924 02:58:18.980738 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:18.980740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:18.980945 (+ 205us) server_negotiation.cc:407] Connection header received
0924 02:58:18.981398 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:18.981399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:18.981411 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:18.981456 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:18.982052 (+ 596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:18.982613 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:18.983527 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:18.983720 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:18.983770 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:18.984389 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:18.984390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:18.984390 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:58:18.984396 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:18.984409 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:58:18.984413 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:18.984450 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:18.984500 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 02:58:18.984528 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 02:58:18.984788 9787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:18.980663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36925 (local address 127.0.98.1:40861)
0924 02:58:18.980838 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:18.980870 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:18.981327 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:18.981515 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:18.981518 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:18.981526 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:18.981987 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:18.981991 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:18.982687 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:18.982688 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:18.983458 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:18.983460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:18.983779 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:18.983780 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:18.983879 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:18.984280 (+ 401us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 02:58:18.984280 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:18.984306 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 02:58:18.984326 (+ 20us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:58:18.984449 (+ 123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:58:18.984451 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:58:18.984453 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 02:58:18.984496 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 02:58:18.984590 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":83,"threads_started":1}
I20260924 02:58:18.985215 9785 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36925
I20260924 02:58:18.985294 9785 heartbeater.cc:464] Registering TS with master...
I20260924 02:58:18.985455 9785 heartbeater.cc:511] Master 127.0.98.62:36925 requested a full tablet report, sending...
I20260924 02:58:18.985731 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 9657
I20260924 02:58:18.985816 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:58:18.986018 392 external_mini_cluster.cc:1428] Setting key b16edc2119a98dd35644da3edf355aa1
I20260924 02:58:18.986006 9597 ts_manager.cc:194] Registered new tserver with Master: 8c3431bd09c145948bf896c053868675 (127.0.98.1:44883)
I20260924 02:58:18.986732 9597 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:40861
I20260924 02:58:18.987192 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36925
--builtin_ntp_servers=127.0.98.20:35129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 02:58:19.064298 9788 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:19.064487 9788 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:19.064507 9788 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:19.065582 9788 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:19.065613 9788 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:19.065635 9788 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:58:19.066931 9788 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9788
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:19.067114 9788 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:19.067349 9788 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:19.072062 9796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:19.072095 9794 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:19.072093 9793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:19.072275 9788 server_base.cc:1034] running on GCE node
I20260924 02:58:19.072463 9788 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:19.072652 9788 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:19.073793 9788 hybrid_clock.cc:648] HybridClock initialized: now 1790218699073778 us; error 26 us; skew 500 ppm
I20260924 02:58:19.075125 9788 webserver.cc:460] Webserver started at http://127.0.98.2:35759/ using document root <none> and password file <none>
I20260924 02:58:19.075341 9788 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:19.075388 9788 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:19.075510 9788 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:58:19.076512 9788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "50ee46659fb845a1932cee3c546435a2"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "7938393f212f968bb42a160910ea76ee"
server_key_iv: "3c636881307bdecc4fb2aceeed8aa409"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.076828 9788 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "50ee46659fb845a1932cee3c546435a2"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "7938393f212f968bb42a160910ea76ee"
server_key_iv: "3c636881307bdecc4fb2aceeed8aa409"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.077968 9788 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:58:19.078584 9802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.078711 9788 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.078788 9788 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "50ee46659fb845a1932cee3c546435a2"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "7938393f212f968bb42a160910ea76ee"
server_key_iv: "3c636881307bdecc4fb2aceeed8aa409"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.078851 9788 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:19.088060 9788 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:19.088316 9788 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:19.088433 9788 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:19.088616 9788 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:58:19.088901 9788 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:58:19.088934 9788 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.088961 9788 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:58:19.088979 9788 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.094170 9788 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:37799
I20260924 02:58:19.094341 9915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:37799 every 8 connection(s)
I20260924 02:58:19.094570 9788 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 02:58:19.099273 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:19.095326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:39319 (local address 127.0.98.62:36925)
0924 02:58:19.095403 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:19.095405 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:19.095577 (+ 172us) server_negotiation.cc:407] Connection header received
0924 02:58:19.096058 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:19.096059 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:19.096069 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:19.096118 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:19.096740 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:19.097237 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:19.098159 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:19.098338 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:19.098387 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:19.099084 (+ 697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:19.099085 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:19.099085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:58:19.099089 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:19.099102 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:58:19.099106 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:19.099144 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:19.099195 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 02:58:19.099216 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260924 02:58:19.099455 9918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:19.095302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36925 (local address 127.0.98.2:39319)
0924 02:58:19.095471 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:19.095501 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:19.095981 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:19.096171 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:19.096174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:19.096183 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:19.096655 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:19.096659 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:19.097300 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:19.097301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:19.098088 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:19.098090 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:19.098393 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:19.098394 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:19.098493 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:19.098969 (+ 476us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 02:58:19.098970 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:19.098996 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 02:58:19.099019 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:58:19.099144 (+ 125us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:58:19.099146 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:58:19.099148 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 02:58:19.099191 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 02:58:19.099265 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":76,"threads_started":1}
I20260924 02:58:19.099880 9916 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36925
I20260924 02:58:19.099961 9916 heartbeater.cc:464] Registering TS with master...
I20260924 02:58:19.100122 9916 heartbeater.cc:511] Master 127.0.98.62:36925 requested a full tablet report, sending...
I20260924 02:58:19.100441 9597 ts_manager.cc:194] Registered new tserver with Master: 50ee46659fb845a1932cee3c546435a2 (127.0.98.2:37799)
I20260924 02:58:19.100785 9597 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:39319
I20260924 02:58:19.101253 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 9788
I20260924 02:58:19.101318 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:58:19.101480 392 external_mini_cluster.cc:1428] Setting key 531213150b05bca19e003c233ac05cc4
I20260924 02:58:19.102221 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36925
--builtin_ntp_servers=127.0.98.20:35129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260924 02:58:19.180161 9919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:19.180346 9919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:19.180365 9919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:19.181524 9919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:19.181569 9919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:19.181596 9919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:58:19.183036 9919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:36925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:19.183254 9919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:19.183516 9919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:19.188201 9925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:19.188210 9924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:19.188257 9919 server_base.cc:1034] running on GCE node
W20260924 02:58:19.188234 9927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:19.188578 9919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:19.188783 9919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:19.189911 9919 hybrid_clock.cc:648] HybridClock initialized: now 1790218699189900 us; error 27 us; skew 500 ppm
I20260924 02:58:19.191114 9919 webserver.cc:460] Webserver started at http://127.0.98.3:35295/ using document root <none> and password file <none>
I20260924 02:58:19.191323 9919 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:19.191370 9919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:19.191488 9919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:58:19.192519 9919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "3e4d6e98d6c44cd7ac8d5f56231200ae"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "641d87b992a537c02a5c04f712d08882"
server_key_iv: "fe4800c85bf8eddb00a3972e706d6515"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.192837 9919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3e4d6e98d6c44cd7ac8d5f56231200ae"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "641d87b992a537c02a5c04f712d08882"
server_key_iv: "fe4800c85bf8eddb00a3972e706d6515"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.193979 9919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:58:19.194619 9933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.194764 9919 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.194842 9919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "3e4d6e98d6c44cd7ac8d5f56231200ae"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "641d87b992a537c02a5c04f712d08882"
server_key_iv: "fe4800c85bf8eddb00a3972e706d6515"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.194905 9919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:19.202466 9919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:19.202759 9919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:19.202872 9919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:19.203068 9919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:58:19.203351 9919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:58:19.203388 9919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.203418 9919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:58:19.203441 9919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.208768 9919 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:37929
I20260924 02:58:19.208822 10046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:37929 every 8 connection(s)
I20260924 02:58:19.209105 9919 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 02:58:19.213963 9637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:19.209714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36445 (local address 127.0.98.62:36925)
0924 02:58:19.209759 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:19.209761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:19.209969 (+ 208us) server_negotiation.cc:407] Connection header received
0924 02:58:19.210435 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:19.210436 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:19.210446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:19.210506 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:19.211399 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:19.211916 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:19.212842 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:19.213020 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:19.213065 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:19.213753 (+ 688us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:19.213754 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:19.213754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:58:19.213759 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:19.213773 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:58:19.213777 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:19.213816 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:19.213878 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 02:58:19.213903 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 02:58:19.214140 10049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:19.209686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:36925 (local address 127.0.98.3:36445)
0924 02:58:19.209855 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:19.209894 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:19.210354 (+ 460us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:19.210638 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:19.210642 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:19.210650 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:19.211210 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:19.211216 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:19.211998 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:19.211999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:19.212782 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:19.212784 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:19.213073 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:19.213074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:19.213174 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:19.213636 (+ 462us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 02:58:19.213636 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:19.213663 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 02:58:19.213684 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:58:19.213821 (+ 137us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:58:19.213824 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:58:19.213826 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 02:58:19.213871 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 02:58:19.213945 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260924 02:58:19.214699 10047 heartbeater.cc:347] Connected to a master server at 127.0.98.62:36925
I20260924 02:58:19.214784 10047 heartbeater.cc:464] Registering TS with master...
I20260924 02:58:19.214949 10047 heartbeater.cc:511] Master 127.0.98.62:36925 requested a full tablet report, sending...
I20260924 02:58:19.215337 9597 ts_manager.cc:194] Registered new tserver with Master: 3e4d6e98d6c44cd7ac8d5f56231200ae (127.0.98.3:37929)
I20260924 02:58:19.215739 9597 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:36445
I20260924 02:58:19.216147 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 9919
I20260924 02:58:19.216235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:58:19.216429 392 external_mini_cluster.cc:1428] Setting key 4e37ad93b88f1dea00762edd38faa2a8
I20260924 02:58:19.217190 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 02:58:19.218883 9637 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:19.218398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43008 (local address 127.0.98.62:36925)
0924 02:58:19.218448 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:19.218450 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:19.218570 (+ 120us) server_negotiation.cc:407] Connection header received
0924 02:58:19.218634 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:19.218635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:19.218646 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:19.218703 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:19.218810 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43008: BlockingRecv error: recv got EOF from 127.0.0.1:43008 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 02:58:19.218935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 9657
I20260924 02:58:19.223474 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 9788
I20260924 02:58:19.226871 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 9919
I20260924 02:58:19.230325 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 9566
2026-09-24T02:58:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (543 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-24T02:58:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:58:19Z Disabled control of system clock
I20260924 02:58:19.253927 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40117
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40117
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 02:58:19.331609 10058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:19.331812 10058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:19.331831 10058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:19.332921 10058 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:58:19.332954 10058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:19.332964 10058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:19.332975 10058 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:58:19.332985 10058 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:58:19.334246 10058 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40117
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40117
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:19.334439 10058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:19.334714 10058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:19.339592 10063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:19.339720 10058 server_base.cc:1034] running on GCE node
W20260924 02:58:19.339599 10064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:19.339674 10066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:19.340030 10058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:19.340298 10058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:19.341444 10058 hybrid_clock.cc:648] HybridClock initialized: now 1790218699341405 us; error 53 us; skew 500 ppm
I20260924 02:58:19.342706 10058 webserver.cc:460] Webserver started at http://127.0.98.62:45749/ using document root <none> and password file <none>
I20260924 02:58:19.342916 10058 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:19.342958 10058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:19.343078 10058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:58:19.344094 10058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "45a6d1ff3af74c38abf41b23ca9e93fd"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "13bcde1febbe3c43d18659d4e2a0a80b"
server_key_iv: "d9d2938ace35896c682589bea74170ff"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.344401 10058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "45a6d1ff3af74c38abf41b23ca9e93fd"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "13bcde1febbe3c43d18659d4e2a0a80b"
server_key_iv: "d9d2938ace35896c682589bea74170ff"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.345614 10058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 02:58:19.346228 10072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.346381 10058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:58:19.346470 10058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "45a6d1ff3af74c38abf41b23ca9e93fd"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "13bcde1febbe3c43d18659d4e2a0a80b"
server_key_iv: "d9d2938ace35896c682589bea74170ff"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.346609 10058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:19.349713 10058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:19.349948 10058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:19.350061 10058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:19.353577 10058 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40117
I20260924 02:58:19.353610 10124 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40117 every 8 connection(s)
I20260924 02:58:19.353885 10058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:58:19.354442 10125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:58:19.357018 10125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd: Bootstrap starting.
I20260924 02:58:19.357550 10125 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd: Neither blocks nor log segments found. Creating new log.
I20260924 02:58:19.357810 10125 log.cc:824] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd: Log is configured to *not* fsync() on all Append() calls
I20260924 02:58:19.358435 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 10058
I20260924 02:58:19.358472 10125 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd: No bootstrap required, opened a new log
I20260924 02:58:19.358575 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:58:19.358769 392 external_mini_cluster.cc:1428] Setting key 3996f435c1941669fbac73fec88a8221
I20260924 02:58:19.360226 10125 raft_consensus.cc:348] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a6d1ff3af74c38abf41b23ca9e93fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40117 } }
I20260924 02:58:19.360353 10125 raft_consensus.cc:374] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:58:19.360383 10125 raft_consensus.cc:729] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45a6d1ff3af74c38abf41b23ca9e93fd, State: Initialized, Role: FOLLOWER
I20260924 02:58:19.360520 10125 consensus_queue.cc:260] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a6d1ff3af74c38abf41b23ca9e93fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40117 } }
I20260924 02:58:19.360596 10125 raft_consensus.cc:388] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:58:19.360626 10125 raft_consensus.cc:482] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:58:19.360667 10125 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:58:19.361192 10125 raft_consensus.cc:504] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a6d1ff3af74c38abf41b23ca9e93fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40117 } }
I20260924 02:58:19.361294 10125 leader_election.cc:302] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 45a6d1ff3af74c38abf41b23ca9e93fd; no voters:
I20260924 02:58:19.361435 10129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:19.359254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60542 (local address 127.0.98.62:40117)
0924 02:58:19.359441 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:19.359448 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:19.359472 (+ 24us) server_negotiation.cc:407] Connection header received
0924 02:58:19.360210 (+ 738us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:19.360216 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:19.360262 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:19.360383 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:19.361012 (+ 629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:19.361015 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:19.361015 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:58:19.361019 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:19.361058 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:58:19.361069 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:19.361123 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:19.361194 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 02:58:19.361249 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260924 02:58:19.361444 10125 leader_election.cc:288] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:58:19.361691 10125 sys_catalog.cc:564] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:58:19.361922 10130 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:58:19.362031 10130 raft_consensus.cc:686] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [term 1 LEADER]: Becoming Leader. State: Replica: 45a6d1ff3af74c38abf41b23ca9e93fd, State: Running, Role: LEADER
I20260924 02:58:19.362111 10130 consensus_queue.cc:237] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a6d1ff3af74c38abf41b23ca9e93fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40117 } }
I20260924 02:58:19.362437 10130 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 45a6d1ff3af74c38abf41b23ca9e93fd. Latest consensus state: current_term: 1 leader_uuid: "45a6d1ff3af74c38abf41b23ca9e93fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a6d1ff3af74c38abf41b23ca9e93fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40117 } } }
I20260924 02:58:19.362444 10132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "45a6d1ff3af74c38abf41b23ca9e93fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "45a6d1ff3af74c38abf41b23ca9e93fd" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40117 } } }
I20260924 02:58:19.362512 10130 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [sys.catalog]: This master's current role is: LEADER
I20260924 02:58:19.362569 10132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd [sys.catalog]: This master's current role is: LEADER
I20260924 02:58:19.362709 10139 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:58:19.363237 10139 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:58:19.364691 10139 catalog_manager.cc:1347] Generated new cluster ID: 0f31249b405d4c7da17e27cd760b2886
I20260924 02:58:19.364741 10139 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:58:19.388849 10139 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:58:19.389046 10139 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:58:19.397519 10139 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 45a6d1ff3af74c38abf41b23ca9e93fd: Generated new TSK 0
I20260924 02:58:19.397735 10139 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 02:58:19.399139 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40117
--builtin_ntp_servers=127.0.98.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 02:58:19.489642 10149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:19.489857 10149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:19.489879 10149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:19.491122 10149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:19.491165 10149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:19.491191 10149 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:58:19.492543 10149 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:19.492770 10149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:19.493053 10149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:19.497885 10154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:19.497850 10155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:19.497848 10157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:19.498260 10149 server_base.cc:1034] running on GCE node
I20260924 02:58:19.498459 10149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:19.498711 10149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:19.499856 10149 hybrid_clock.cc:648] HybridClock initialized: now 1790218699499842 us; error 29 us; skew 500 ppm
I20260924 02:58:19.501082 10149 webserver.cc:460] Webserver started at http://127.0.98.1:45871/ using document root <none> and password file <none>
I20260924 02:58:19.501298 10149 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:19.501346 10149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:19.501480 10149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:58:19.502568 10149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "86fdab0086d8448a82ce00e8568016d6"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "4158442289af5a052450735e98af7676"
server_key_iv: "1c2058a6cbee8ed063b2c202df44a76f"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.502898 10149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "86fdab0086d8448a82ce00e8568016d6"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "4158442289af5a052450735e98af7676"
server_key_iv: "1c2058a6cbee8ed063b2c202df44a76f"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.504127 10149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 02:58:19.504817 10163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.504997 10149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:58:19.505110 10149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "86fdab0086d8448a82ce00e8568016d6"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "4158442289af5a052450735e98af7676"
server_key_iv: "1c2058a6cbee8ed063b2c202df44a76f"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.505169 10149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:19.508477 10149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:19.508734 10149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:19.508853 10149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:19.509037 10149 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:58:19.509326 10149 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:58:19.509356 10149 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.509382 10149 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:58:19.509406 10149 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.514636 10149 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:32779
I20260924 02:58:19.514694 10276 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:32779 every 8 connection(s)
I20260924 02:58:19.514976 10149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:58:19.517740 10129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:19.515618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49517 (local address 127.0.98.62:40117)
0924 02:58:19.515679 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:19.515681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:19.515879 (+ 198us) server_negotiation.cc:407] Connection header received
0924 02:58:19.516386 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:19.516387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:19.516395 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:19.516445 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:19.517498 (+ 1053us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:19.517500 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:19.517500 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:58:19.517505 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:19.517522 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:58:19.517526 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:19.517575 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:19.517656 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 02:58:19.517672 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 02:58:19.517918 10279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:19.515599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40117 (local address 127.0.98.1:49517)
0924 02:58:19.515765 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:19.515796 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:19.516286 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:19.516514 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:19.516518 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:19.516528 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:19.517354 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 02:58:19.517355 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:19.517382 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 02:58:19.517411 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:58:19.517577 (+ 166us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:58:19.517582 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:58:19.517585 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:58:19.517656 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 02:58:19.517717 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":72,"threads_started":1}
I20260924 02:58:19.518363 10277 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40117
I20260924 02:58:19.518435 10277 heartbeater.cc:464] Registering TS with master...
I20260924 02:58:19.518602 10277 heartbeater.cc:511] Master 127.0.98.62:40117 requested a full tablet report, sending...
I20260924 02:58:19.519048 10089 ts_manager.cc:194] Registered new tserver with Master: 86fdab0086d8448a82ce00e8568016d6 (127.0.98.1:32779)
I20260924 02:58:19.523988 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 10149
I20260924 02:58:19.524056 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:58:19.524206 392 external_mini_cluster.cc:1428] Setting key 6b726e08a385702f0e7a5974b2855c5c
I20260924 02:58:19.524989 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40117
--builtin_ntp_servers=127.0.98.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 02:58:19.601928 10280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:19.602123 10280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:19.602150 10280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:19.603305 10280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:19.603345 10280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:19.603371 10280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:58:19.604641 10280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:19.604846 10280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:19.605093 10280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:19.609692 10288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:19.609705 10286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:19.609798 10280 server_base.cc:1034] running on GCE node
W20260924 02:58:19.609900 10285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:19.610093 10280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:19.610289 10280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:19.611450 10280 hybrid_clock.cc:648] HybridClock initialized: now 1790218699611437 us; error 25 us; skew 500 ppm
I20260924 02:58:19.612581 10280 webserver.cc:460] Webserver started at http://127.0.98.2:35147/ using document root <none> and password file <none>
I20260924 02:58:19.612784 10280 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:19.612833 10280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:19.612951 10280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:58:19.613958 10280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "30aea6468e7241d2a31ad3eb273229a6"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "8976238da79cae0c14b6bcb39bb4473d"
server_key_iv: "ad1fac71507b475925455726c1d8d0b5"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.614274 10280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "30aea6468e7241d2a31ad3eb273229a6"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "8976238da79cae0c14b6bcb39bb4473d"
server_key_iv: "ad1fac71507b475925455726c1d8d0b5"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.615464 10280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 02:58:19.616086 10294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.616228 10280 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:58:19.616304 10280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "30aea6468e7241d2a31ad3eb273229a6"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "8976238da79cae0c14b6bcb39bb4473d"
server_key_iv: "ad1fac71507b475925455726c1d8d0b5"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.616366 10280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:19.619535 10280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:19.619760 10280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:19.619870 10280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:19.620049 10280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:58:19.620306 10280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:58:19.620337 10280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.620361 10280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:58:19.620385 10280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.625581 10280 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39063
I20260924 02:58:19.625615 10407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39063 every 8 connection(s)
I20260924 02:58:19.625886 10280 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 02:58:19.628633 10129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:19.626565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:54283 (local address 127.0.98.62:40117)
0924 02:58:19.626621 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:19.626623 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:19.626790 (+ 167us) server_negotiation.cc:407] Connection header received
0924 02:58:19.627268 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:19.627269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:19.627276 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:19.627334 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:19.628398 (+ 1064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:19.628400 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:19.628400 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:58:19.628404 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:19.628421 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:58:19.628425 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:19.628476 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:19.628551 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 02:58:19.628564 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 02:58:19.628798 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:19.626495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40117 (local address 127.0.98.2:54283)
0924 02:58:19.626680 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:19.626715 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:19.627172 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:19.627410 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:19.627414 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:19.627423 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:19.628261 (+ 838us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 02:58:19.628262 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:19.628287 (+ 25us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 02:58:19.628313 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:58:19.628484 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:58:19.628489 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:58:19.628491 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 02:58:19.628556 (+ 65us) client_negotiation.cc:240] Negotiation successful
0924 02:58:19.628614 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260924 02:58:19.629204 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 10280
I20260924 02:58:19.629271 10408 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40117
I20260924 02:58:19.629298 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:58:19.629357 10408 heartbeater.cc:464] Registering TS with master...
I20260924 02:58:19.629469 392 external_mini_cluster.cc:1428] Setting key a35c09a78db684263e9c9699b19e6d17
I20260924 02:58:19.629495 10408 heartbeater.cc:511] Master 127.0.98.62:40117 requested a full tablet report, sending...
I20260924 02:58:19.629833 10088 ts_manager.cc:194] Registered new tserver with Master: 30aea6468e7241d2a31ad3eb273229a6 (127.0.98.2:39063)
I20260924 02:58:19.630330 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40117
--builtin_ntp_servers=127.0.98.20:32887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260924 02:58:19.707863 10411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:19.708088 10411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:19.708119 10411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:19.709275 10411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:19.709309 10411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:19.709343 10411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:58:19.710704 10411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40117
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:19.710922 10411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:19.711185 10411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:19.715711 10419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:19.715785 10417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:19.715763 10416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:19.715919 10411 server_base.cc:1034] running on GCE node
I20260924 02:58:19.716101 10411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:19.716311 10411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:19.717473 10411 hybrid_clock.cc:648] HybridClock initialized: now 1790218699717459 us; error 28 us; skew 500 ppm
I20260924 02:58:19.718719 10411 webserver.cc:460] Webserver started at http://127.0.98.3:42799/ using document root <none> and password file <none>
I20260924 02:58:19.718927 10411 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:19.718973 10411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:19.719100 10411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:58:19.720121 10411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "3af35a3f291a440dbd8410298dd2df32"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "122fa60fa43e01d524e94325cdcb94aa"
server_key_iv: "74f33c7f91571dfa7fa42b9c95e4fc37"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.720424 10411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "3af35a3f291a440dbd8410298dd2df32"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "122fa60fa43e01d524e94325cdcb94aa"
server_key_iv: "74f33c7f91571dfa7fa42b9c95e4fc37"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.721555 10411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 02:58:19.722198 10425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.722343 10411 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:58:19.722417 10411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "3af35a3f291a440dbd8410298dd2df32"
format_stamp: "Formatted at 2026-09-24 02:58:19 on dist-test-slave-sb6p"
server_key: "122fa60fa43e01d524e94325cdcb94aa"
server_key_iv: "74f33c7f91571dfa7fa42b9c95e4fc37"
server_key_version: "encryptionkey@0"
I20260924 02:58:19.722474 10411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:19.725591 10411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:19.725811 10411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:19.725922 10411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:19.726092 10411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:58:19.726361 10411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:58:19.726392 10411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.726415 10411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:58:19.726451 10411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:19.731681 10411 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:33115
I20260924 02:58:19.731729 10538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:33115 every 8 connection(s)
I20260924 02:58:19.731995 10411 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 02:58:19.734277 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 10411
I20260924 02:58:19.734367 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:58:19.734607 392 external_mini_cluster.cc:1428] Setting key 38058c258e142bff0ec3690fe7e1be80
I20260924 02:58:19.734651 10129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:19.732565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:48749 (local address 127.0.98.62:40117)
0924 02:58:19.732616 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:19.732618 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:19.732833 (+ 215us) server_negotiation.cc:407] Connection header received
0924 02:58:19.733269 (+ 436us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:19.733270 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:19.733275 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:19.733310 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:19.734389 (+ 1079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:19.734390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:19.734391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 02:58:19.734395 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:19.734410 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 02:58:19.734415 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:19.734467 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:19.734585 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 02:58:19.734599 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 02:58:19.734853 10541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:19.732557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40117 (local address 127.0.98.3:48749)
0924 02:58:19.732719 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:19.732756 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:19.733179 (+ 423us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:19.733368 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:19.733371 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:19.733380 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:19.734219 (+ 839us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 02:58:19.734220 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:19.734260 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 02:58:19.734294 (+ 34us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:58:19.734480 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:58:19.734486 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:58:19.734489 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:58:19.734593 (+ 104us) client_negotiation.cc:240] Negotiation successful
0924 02:58:19.734655 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":64,"threads_started":1}
I20260924 02:58:19.735213 10539 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40117
I20260924 02:58:19.735291 10539 heartbeater.cc:464] Registering TS with master...
I20260924 02:58:19.735419 10539 heartbeater.cc:511] Master 127.0.98.62:40117 requested a full tablet report, sending...
I20260924 02:58:19.735744 10089 ts_manager.cc:194] Registered new tserver with Master: 3af35a3f291a440dbd8410298dd2df32 (127.0.98.3:33115)
I20260924 02:58:19.746258 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260924 02:58:19.748096 10129 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:58:19.747571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.0.98.62:40117)
0924 02:58:19.747625 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:19.747627 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:19.747769 (+ 142us) server_negotiation.cc:407] Connection header received
0924 02:58:19.747840 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:19.747841 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:19.747847 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:19.747888 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:19.748026 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60558: BlockingRecv error: recv got EOF from 127.0.0.1:60558 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 02:58:19.748142 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 10149
I20260924 02:58:19.752323 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 10280
I20260924 02:58:19.755571 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 10411
I20260924 02:58:19.759286 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 10058
2026-09-24T02:58:19Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (529 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:58:19 dist-test-slave-sb6p krb5kdc[10549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:58:19 dist-test-slave-sb6p krb5kdc[10549](info): set up 2 sockets
Sep 24 02:58:19 dist-test-slave-sb6p krb5kdc[10549](info): commencing operation
krb5kdc: starting...
W20260924 02:58:20.711306 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:58:20 dist-test-slave-sb6p krb5kdc[10549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:58:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:58:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 02:58:20.819132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34085
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34085
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:58:20.897420 10562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:20.897624 10562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:20.897644 10562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:20.898746 10562 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:58:20.898779 10562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:20.898792 10562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:20.898802 10562 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:58:20.898815 10562 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:58:20.900190 10562 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34085
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34085
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10562
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:20.900377 10562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:20.900556 10562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:20.905119 10567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:20.905220 10562 server_base.cc:1034] running on GCE node
W20260924 02:58:20.905110 10568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:20.905112 10570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:20.905534 10562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:20.905819 10562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:20.906949 10562 hybrid_clock.cc:648] HybridClock initialized: now 1790218700906935 us; error 27 us; skew 500 ppm
Sep 24 02:58:20 dist-test-slave-sb6p krb5kdc[10549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218700, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:58:20.909162 10562 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:58:20.909539 10562 webserver.cc:460] Webserver started at http://127.0.98.62:41493/ using document root <none> and password file <none>
I20260924 02:58:20.909771 10562 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:20.909819 10562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:20.909935 10562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:58:20.910967 10562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "bdee9f0fd7674ec79fc27f9eec787387"
format_stamp: "Formatted at 2026-09-24 02:58:20 on dist-test-slave-sb6p"
server_key: "f54cab20afec391c907e9770b8279d0f"
server_key_iv: "19342ff39581716074fd5109c0e287cb"
server_key_version: "encryptionkey@0"
I20260924 02:58:20.911268 10562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "bdee9f0fd7674ec79fc27f9eec787387"
format_stamp: "Formatted at 2026-09-24 02:58:20 on dist-test-slave-sb6p"
server_key: "f54cab20afec391c907e9770b8279d0f"
server_key_iv: "19342ff39581716074fd5109c0e287cb"
server_key_version: "encryptionkey@0"
I20260924 02:58:20.912415 10562 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:58:20.913093 10577 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:20.913264 10562 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:58:20.913340 10562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "bdee9f0fd7674ec79fc27f9eec787387"
format_stamp: "Formatted at 2026-09-24 02:58:20 on dist-test-slave-sb6p"
server_key: "f54cab20afec391c907e9770b8279d0f"
server_key_iv: "19342ff39581716074fd5109c0e287cb"
server_key_version: "encryptionkey@0"
I20260924 02:58:20.913396 10562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:20.936683 10562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:20.938457 10562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:20.938630 10562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:20.942142 10562 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34085
I20260924 02:58:20.942181 10629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34085 every 8 connection(s)
I20260924 02:58:20.942485 10562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:58:20.943322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 10562
I20260924 02:58:20.943393 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:58:20.943544 10630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:58:20.943568 392 external_mini_cluster.cc:1428] Setting key df66810a85c61336ba54bd5a920db725
I20260924 02:58:20.946832 10630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387: Bootstrap starting.
I20260924 02:58:20.947389 10630 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387: Neither blocks nor log segments found. Creating new log.
I20260924 02:58:20.947670 10630 log.cc:824] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387: Log is configured to *not* fsync() on all Append() calls
I20260924 02:58:20.948346 10630 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387: No bootstrap required, opened a new log
Sep 24 02:58:20 dist-test-slave-sb6p krb5kdc[10549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218700, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:58:20.950152 10630 raft_consensus.cc:348] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdee9f0fd7674ec79fc27f9eec787387" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } }
I20260924 02:58:20.950325 10630 raft_consensus.cc:374] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:58:20.950371 10630 raft_consensus.cc:729] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bdee9f0fd7674ec79fc27f9eec787387, State: Initialized, Role: FOLLOWER
I20260924 02:58:20.950505 10630 consensus_queue.cc:260] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdee9f0fd7674ec79fc27f9eec787387" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } }
I20260924 02:58:20.950627 10630 raft_consensus.cc:388] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:58:20.950661 10630 raft_consensus.cc:482] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:58:20.950726 10630 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:58:20.951203 10630 raft_consensus.cc:504] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdee9f0fd7674ec79fc27f9eec787387" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } }
I20260924 02:58:20.951285 10630 leader_election.cc:302] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bdee9f0fd7674ec79fc27f9eec787387; no voters:
I20260924 02:58:20.951426 10630 leader_election.cc:288] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:58:20.951465 10635 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:58:20.951594 10635 raft_consensus.cc:686] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [term 1 LEADER]: Becoming Leader. State: Replica: bdee9f0fd7674ec79fc27f9eec787387, State: Running, Role: LEADER
I20260924 02:58:20.951683 10630 sys_catalog.cc:564] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:58:20.951725 10635 consensus_queue.cc:237] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdee9f0fd7674ec79fc27f9eec787387" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } }
I20260924 02:58:20.952059 10635 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bdee9f0fd7674ec79fc27f9eec787387. Latest consensus state: current_term: 1 leader_uuid: "bdee9f0fd7674ec79fc27f9eec787387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdee9f0fd7674ec79fc27f9eec787387" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } } }
I20260924 02:58:20.952164 10635 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [sys.catalog]: This master's current role is: LEADER
I20260924 02:58:20.952107 10637 sys_catalog.cc:455] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bdee9f0fd7674ec79fc27f9eec787387" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bdee9f0fd7674ec79fc27f9eec787387" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34085 } } }
I20260924 02:58:20.952325 10637 sys_catalog.cc:458] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387 [sys.catalog]: This master's current role is: LEADER
I20260924 02:58:20.952383 10644 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:58:20.952862 10644 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:58:20.953702 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:20.944060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60558 (local address 127.0.98.62:34085)
0924 02:58:20.944262 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:20.944272 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:20.944305 (+ 33us) server_negotiation.cc:407] Connection header received
0924 02:58:20.944687 (+ 382us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:20.944696 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:20.944759 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:20.944909 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:20.945447 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:20.946786 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:20.947521 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:20.948059 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:20.948172 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:20.950196 (+ 2024us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:20.950199 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:20.950200 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:58:20.950211 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:20.952010 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:20.952375 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:20.952378 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:20.952379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:20.952501 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:20.952698 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:20.952699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:20.952700 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:20.953003 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:20.953205 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:20.953408 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 02:58:20.953532 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":106,"threads_started":1}
I20260924 02:58:20.954993 10644 catalog_manager.cc:1347] Generated new cluster ID: de90112115ba43afa840aa9a465af61f
I20260924 02:58:20.955036 10644 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:58:20.969950 10644 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:58:20.970645 10644 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:58:20.973691 10644 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387: Generated new TSK 0
I20260924 02:58:20.973851 10644 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 02:58:20.999265 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34085
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:58:21.076835 10656 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:21.077024 10656 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:21.077045 10656 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:21.078183 10656 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:21.078222 10656 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:21.078248 10656 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:58:21.079686 10656 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10656
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:21.079905 10656 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:21.080109 10656 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 02:58:21.084797 10656 server_base.cc:1034] running on GCE node
W20260924 02:58:21.084731 10662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:21.084736 10661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:21.084761 10664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:21.085124 10656 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:21.085340 10656 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:21.086483 10656 hybrid_clock.cc:648] HybridClock initialized: now 1790218701086469 us; error 27 us; skew 500 ppm
Sep 24 02:58:21 dist-test-slave-sb6p krb5kdc[10549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:58:21.088766 10656 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 02:58:21.089149 10656 webserver.cc:460] Webserver started at http://127.0.98.1:38533/ using document root <none> and password file <none>
I20260924 02:58:21.089339 10656 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:21.089385 10656 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:21.089502 10656 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:58:21.090507 10656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "d47e43eedcf5407b9b3dfc0cfd77de7c"
format_stamp: "Formatted at 2026-09-24 02:58:21 on dist-test-slave-sb6p"
server_key: "e6838a5ecc0624c2a6bf2597212a4676"
server_key_iv: "570fa2c9c4c4e02846467613f83161b7"
server_key_version: "encryptionkey@0"
I20260924 02:58:21.090862 10656 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "d47e43eedcf5407b9b3dfc0cfd77de7c"
format_stamp: "Formatted at 2026-09-24 02:58:21 on dist-test-slave-sb6p"
server_key: "e6838a5ecc0624c2a6bf2597212a4676"
server_key_iv: "570fa2c9c4c4e02846467613f83161b7"
server_key_version: "encryptionkey@0"
I20260924 02:58:21.092033 10656 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 02:58:21.092685 10671 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:21.092823 10656 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:58:21.092924 10656 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "d47e43eedcf5407b9b3dfc0cfd77de7c"
format_stamp: "Formatted at 2026-09-24 02:58:21 on dist-test-slave-sb6p"
server_key: "e6838a5ecc0624c2a6bf2597212a4676"
server_key_iv: "570fa2c9c4c4e02846467613f83161b7"
server_key_version: "encryptionkey@0"
I20260924 02:58:21.092981 10656 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:21.105724 10656 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:21.107626 10656 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:21.107754 10656 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:21.107955 10656 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:58:21.108228 10656 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:58:21.108259 10656 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:21.108289 10656 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:58:21.108309 10656 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:21.113679 10656 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37791
I20260924 02:58:21.113735 10784 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37791 every 8 connection(s)
I20260924 02:58:21.114014 10656 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 02:58:21 dist-test-slave-sb6p krb5kdc[10549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:58:21.122706 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.114814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52673 (local address 127.0.98.62:34085)
0924 02:58:21.114868 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:21.114870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:21.115020 (+ 150us) server_negotiation.cc:407] Connection header received
0924 02:58:21.115482 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:21.115483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:21.115491 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:21.115560 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:21.116476 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.117054 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.118016 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.118219 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.118274 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.120259 (+ 1985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:21.120261 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:21.120261 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:58:21.120269 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:21.121469 (+ 1200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:21.121823 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:21.121824 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:21.121824 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:21.121874 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:21.122088 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:21.122088 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:21.122089 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:21.122191 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:21.122247 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:21.122497 (+ 250us) server_negotiation.cc:299] Negotiation successful
0924 02:58:21.122617 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 02:58:21.122853 10787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.114744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34085 (local address 127.0.98.1:52673)
0924 02:58:21.114899 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:21.114940 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:21.115410 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:21.115638 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:21.115643 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:21.116215 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:21.116403 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:21.116408 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.117136 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.117137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:21.117941 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:21.117943 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.118279 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.118280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:21.118376 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.118811 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:21.118820 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:21.120169 (+ 1349us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:58:21.121542 (+ 1373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:58:21.121548 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:58:21.121550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:58:21.121759 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:58:21.121926 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:58:21.121926 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:58:21.121927 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:58:21.122033 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:58:21.122246 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:58:21.122251 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:58:21.122384 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 02:58:21.122470 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 02:58:21.122648 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":68,"threads_started":1}
I20260924 02:58:21.123320 10785 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34085
I20260924 02:58:21.123371 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 10656
I20260924 02:58:21.123407 10785 heartbeater.cc:464] Registering TS with master...
I20260924 02:58:21.123458 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:58:21.123579 10785 heartbeater.cc:511] Master 127.0.98.62:34085 requested a full tablet report, sending...
I20260924 02:58:21.123688 392 external_mini_cluster.cc:1428] Setting key cca9a074e62c0ee88c950fbd0b006c5c
I20260924 02:58:21.124083 10594 ts_manager.cc:194] Registered new tserver with Master: d47e43eedcf5407b9b3dfc0cfd77de7c (127.0.98.1:37791)
I20260924 02:58:21.124779 10594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52673
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 02:58:21.148931 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34085
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:58:21.228191 10790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:21.228391 10790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:21.228415 10790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:21.229602 10790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:21.229643 10790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:21.229669 10790 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:58:21.231091 10790 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10790
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:21.231302 10790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:21.231511 10790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:21.236172 10795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:21.236181 10798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:21.236192 10796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:21.236209 10790 server_base.cc:1034] running on GCE node
I20260924 02:58:21.236544 10790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:21.236776 10790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:21.237910 10790 hybrid_clock.cc:648] HybridClock initialized: now 1790218701237899 us; error 30 us; skew 500 ppm
Sep 24 02:58:21 dist-test-slave-sb6p krb5kdc[10549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:58:21.240279 10790 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 02:58:21.240674 10790 webserver.cc:460] Webserver started at http://127.0.98.2:37253/ using document root <none> and password file <none>
I20260924 02:58:21.240890 10790 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:21.240939 10790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:21.241087 10790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:58:21.242096 10790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "e074a71a3f3a412c906e226df177b4fb"
format_stamp: "Formatted at 2026-09-24 02:58:21 on dist-test-slave-sb6p"
server_key: "f99ffb9e13257fc15512b157e270acc3"
server_key_iv: "a28cd3c997c7dbea884976fa156a6ad5"
server_key_version: "encryptionkey@0"
I20260924 02:58:21.242416 10790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "e074a71a3f3a412c906e226df177b4fb"
format_stamp: "Formatted at 2026-09-24 02:58:21 on dist-test-slave-sb6p"
server_key: "f99ffb9e13257fc15512b157e270acc3"
server_key_iv: "a28cd3c997c7dbea884976fa156a6ad5"
server_key_version: "encryptionkey@0"
I20260924 02:58:21.243623 10790 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 02:58:21.244247 10805 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:21.244396 10790 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:58:21.244472 10790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "e074a71a3f3a412c906e226df177b4fb"
format_stamp: "Formatted at 2026-09-24 02:58:21 on dist-test-slave-sb6p"
server_key: "f99ffb9e13257fc15512b157e270acc3"
server_key_iv: "a28cd3c997c7dbea884976fa156a6ad5"
server_key_version: "encryptionkey@0"
I20260924 02:58:21.244532 10790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:21.261162 10790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:21.262943 10790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:21.263085 10790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:21.263306 10790 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:58:21.263585 10790 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:58:21.263616 10790 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:21.263644 10790 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:58:21.263667 10790 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:21.269120 10790 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:44885
I20260924 02:58:21.269157 10918 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:44885 every 8 connection(s)
I20260924 02:58:21.269447 10790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 02:58:21.273335 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 10790
I20260924 02:58:21.273428 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:58:21.273624 392 external_mini_cluster.cc:1428] Setting key d3b5d1b4390f55eb7f389b7dc85a86e9
Sep 24 02:58:21 dist-test-slave-sb6p krb5kdc[10549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:58:21.278282 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.270072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:40281 (local address 127.0.98.62:34085)
0924 02:58:21.270129 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:21.270132 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:21.270304 (+ 172us) server_negotiation.cc:407] Connection header received
0924 02:58:21.270821 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:21.270822 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:21.270832 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:21.270878 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:21.271715 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.272207 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.273137 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.273325 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.273377 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.275655 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:21.275657 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:21.275657 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:58:21.275666 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:21.277046 (+ 1380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:21.277399 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:21.277400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:21.277400 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:21.277459 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:21.277699 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:21.277700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:21.277700 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:21.277809 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:21.277866 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:21.278117 (+ 251us) server_negotiation.cc:299] Negotiation successful
0924 02:58:21.278197 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 02:58:21.278443 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.270046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34085 (local address 127.0.98.2:40281)
0924 02:58:21.270198 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:21.270232 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:21.270741 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:21.270930 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:21.270933 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:21.271465 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:21.271651 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:21.271656 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.272270 (+ 614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.272271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:21.273058 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:21.273060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.273402 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.273403 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:21.273539 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.274014 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:21.274023 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:21.275536 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:58:21.277132 (+ 1596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:58:21.277138 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:58:21.277139 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:58:21.277337 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:58:21.277531 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:58:21.277532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:58:21.277532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:58:21.277634 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:58:21.277864 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:58:21.277869 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:58:21.278002 (+ 133us) client_negotiation.cc:769] Sending connection context
0924 02:58:21.278089 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 02:58:21.278237 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":94,"threads_started":1}
I20260924 02:58:21.278885 10919 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34085
I20260924 02:58:21.278985 10919 heartbeater.cc:464] Registering TS with master...
I20260924 02:58:21.279175 10919 heartbeater.cc:511] Master 127.0.98.62:34085 requested a full tablet report, sending...
I20260924 02:58:21.279502 10594 ts_manager.cc:194] Registered new tserver with Master: e074a71a3f3a412c906e226df177b4fb (127.0.98.2:44885)
I20260924 02:58:21.279887 10594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:40281
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 02:58:21.296384 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34085
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:41845
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:58:21.401943 10924 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:58:21.402205 10924 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:58:21.402238 10924 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:58:21.403963 10924 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:58:21.404021 10924 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:58:21.404059 10924 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:58:21.406173 10924 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41845
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34085
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10924
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:58:21.406421 10924 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:58:21.406728 10924 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:58:21.411741 10930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:58:21.411746 10932 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:21.411823 10924 server_base.cc:1034] running on GCE node
W20260924 02:58:21.411757 10929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:58:21.412118 10924 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:58:21.412307 10924 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:58:21.413496 10924 hybrid_clock.cc:648] HybridClock initialized: now 1790218701413435 us; error 27 us; skew 500 ppm
Sep 24 02:58:21 dist-test-slave-sb6p krb5kdc[10549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:58:21.415869 10924 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 02:58:21.416254 10924 webserver.cc:460] Webserver started at http://127.0.98.3:37081/ using document root <none> and password file <none>
I20260924 02:58:21.416456 10924 fs_manager.cc:359] Metadata directory not provided
I20260924 02:58:21.416510 10924 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:58:21.416656 10924 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:58:21.417697 10924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "045852134eb14c4186a96dd680cd88b0"
format_stamp: "Formatted at 2026-09-24 02:58:21 on dist-test-slave-sb6p"
server_key: "d0361a925d41b348137ba329171855ed"
server_key_iv: "3a39740e45b829ac7036d1cae69d5143"
server_key_version: "encryptionkey@0"
I20260924 02:58:21.418011 10924 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "045852134eb14c4186a96dd680cd88b0"
format_stamp: "Formatted at 2026-09-24 02:58:21 on dist-test-slave-sb6p"
server_key: "d0361a925d41b348137ba329171855ed"
server_key_iv: "3a39740e45b829ac7036d1cae69d5143"
server_key_version: "encryptionkey@0"
I20260924 02:58:21.419304 10924 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 02:58:21.420015 10939 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:21.420161 10924 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:58:21.420238 10924 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "045852134eb14c4186a96dd680cd88b0"
format_stamp: "Formatted at 2026-09-24 02:58:21 on dist-test-slave-sb6p"
server_key: "d0361a925d41b348137ba329171855ed"
server_key_iv: "3a39740e45b829ac7036d1cae69d5143"
server_key_version: "encryptionkey@0"
I20260924 02:58:21.420296 10924 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:58:21.432430 10924 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:58:21.434129 10924 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:58:21.434252 10924 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:58:21.434459 10924 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:58:21.434758 10924 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:58:21.434792 10924 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:21.434819 10924 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:58:21.434844 10924 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:58:21.440305 10924 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44199
I20260924 02:58:21.440358 11052 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44199 every 8 connection(s)
I20260924 02:58:21.440629 10924 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 02:58:21.440881 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 10924
I20260924 02:58:21.440979 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:58:21.441181 392 external_mini_cluster.cc:1428] Setting key fa1c30b8776b9962395189033d327fc7
Sep 24 02:58:21 dist-test-slave-sb6p krb5kdc[10549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218701, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:58:21.449591 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.441423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41047 (local address 127.0.98.62:34085)
0924 02:58:21.441468 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:21.441471 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:21.441703 (+ 232us) server_negotiation.cc:407] Connection header received
0924 02:58:21.442265 (+ 562us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:21.442266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:21.442278 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:21.442322 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:21.443299 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.443802 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.444755 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.444958 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.445011 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.447130 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:21.447131 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:21.447132 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:58:21.447138 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:21.448224 (+ 1086us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:21.448597 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:21.448598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:21.448598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:21.448647 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:21.448998 (+ 351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:21.448999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:21.448999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:21.449116 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:21.449179 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:21.449432 (+ 253us) server_negotiation.cc:299] Negotiation successful
0924 02:58:21.449490 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 02:58:21.449745 11055 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.441423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34085 (local address 127.0.98.3:41047)
0924 02:58:21.441579 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:21.441625 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:21.442168 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:21.442400 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:21.442404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:21.443022 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:58:21.443213 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:21.443217 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.443882 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.443883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:21.444677 (+ 794us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:21.444679 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.445030 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.445031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:21.445163 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.445565 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:58:21.445573 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:58:21.447034 (+ 1461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:58:21.448305 (+ 1271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:58:21.448310 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:58:21.448312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:58:21.448541 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:58:21.448797 (+ 256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:58:21.448798 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:58:21.448799 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:58:21.448907 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:58:21.449200 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:58:21.449205 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:58:21.449322 (+ 117us) client_negotiation.cc:769] Sending connection context
0924 02:58:21.449413 (+ 91us) client_negotiation.cc:240] Negotiation successful
0924 02:58:21.449528 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":81,"threads_started":1}
I20260924 02:58:21.450163 11053 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34085
I20260924 02:58:21.450248 11053 heartbeater.cc:464] Registering TS with master...
I20260924 02:58:21.450409 11053 heartbeater.cc:511] Master 127.0.98.62:34085 requested a full tablet report, sending...
I20260924 02:58:21.450806 10594 ts_manager.cc:194] Registered new tserver with Master: 045852134eb14c4186a96dd680cd88b0 (127.0.98.3:44199)
I20260924 02:58:21.451198 10594 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41047
I20260924 02:58:21.452785 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 02:58:21.459228 10633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.454032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60560 (local address 127.0.98.62:34085)
0924 02:58:21.454083 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:21.454104 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:21.454113 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:58:21.454172 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:21.454173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:21.454181 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:21.454224 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:58:21.454744 (+ 520us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.455210 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.455893 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.456082 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.456139 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.457165 (+ 1026us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:58:21.457166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:58:21.457167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:58:21.457173 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:58:21.458237 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:21.458530 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:21.458532 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:21.458532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:21.458583 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:58:21.458747 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:58:21.458748 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:58:21.458749 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:58:21.458884 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:58:21.458970 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:21.459065 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 02:58:21.459143 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 02:58:21.460615 10594 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 02:58:21.462109 10594 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:58:21.467885 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.464083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37791 (local address 127.0.0.1:34128)
0924 02:58:21.464240 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:21.464278 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:21.464338 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:21.464689 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:21.464692 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:21.464694 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:58:21.464761 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:21.464762 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.465677 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.465678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:21.466791 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:21.466793 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.467612 (+ 819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.467613 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:21.467685 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.467698 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 02:58:21.467759 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 02:58:21.467810 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":56,"threads_started":1}
I20260924 02:58:21.467903 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.464253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34128 (local address 127.0.98.1:37791)
0924 02:58:21.464486 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:21.464491 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:21.464501 (+ 10us) server_negotiation.cc:407] Connection header received
0924 02:58:21.464588 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:21.464592 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:21.464610 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:21.464683 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:58:21.464837 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.465600 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.466893 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.467532 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.467597 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.467674 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:21.467780 (+ 106us) server_negotiation.cc:299] Negotiation successful
0924 02:58:21.467815 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":123,"threads_started":1}
I20260924 02:58:21.468259 11063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.464083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:44885 (local address 127.0.0.1:60736)
0924 02:58:21.464542 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:21.464551 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:21.464613 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:21.464826 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:21.464827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:21.464828 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:58:21.464874 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:21.464875 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.466145 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.466146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:21.467064 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:21.467066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.467919 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.467920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:21.468104 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.468108 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:58:21.468151 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 02:58:21.468190 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"mutex_wait_us":23,"thread_start_us":55,"threads_started":1}
I20260924 02:58:21.468371 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.464318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44566 (local address 127.0.98.3:44199)
0924 02:58:21.464566 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:21.464570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:21.464576 (+ 6us) server_negotiation.cc:407] Connection header received
0924 02:58:21.464621 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:21.464623 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:21.464638 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:21.464901 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:58:21.464967 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.466039 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.467349 (+ 1310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.468005 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.468104 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.468156 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:21.468255 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 02:58:21.468289 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":70,"threads_started":1}
I20260924 02:58:21.468500 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.464340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60736 (local address 127.0.98.2:44885)
0924 02:58:21.464499 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:21.464502 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:21.464626 (+ 124us) server_negotiation.cc:407] Connection header received
0924 02:58:21.464673 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:21.464675 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:21.464689 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:21.464754 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:58:21.464940 (+ 186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.466032 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.467137 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.467771 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.467835 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.467885 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:21.468387 (+ 502us) server_negotiation.cc:299] Negotiation successful
0924 02:58:21.468427 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":67,"threads_started":1}
I20260924 02:58:21.468341 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.464098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44199 (local address 127.0.0.1:44566)
0924 02:58:21.464447 (+ 349us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:21.464455 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:21.464491 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:21.464784 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:21.464785 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:21.464787 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:58:21.464830 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:21.464831 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.466131 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.466132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:21.467275 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:21.467277 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.468115 (+ 838us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.468116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:21.468178 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.468182 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:58:21.468242 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 02:58:21.468280 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"mutex_wait_us":40,"thread_start_us":68,"threads_started":1}
I20260924 02:58:21.468753 10719 tablet_service.cc:1467] Processing CreateTablet for tablet 431a014c633d4712979a338065b98fdb (DEFAULT_TABLE table=test-table [id=6f5fdca4501b483abd3b66bf98e6cef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:58:21.469079 10719 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431a014c633d4712979a338065b98fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:58:21.469204 10987 tablet_service.cc:1467] Processing CreateTablet for tablet 431a014c633d4712979a338065b98fdb (DEFAULT_TABLE table=test-table [id=6f5fdca4501b483abd3b66bf98e6cef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:58:21.469211 10853 tablet_service.cc:1467] Processing CreateTablet for tablet 431a014c633d4712979a338065b98fdb (DEFAULT_TABLE table=test-table [id=6f5fdca4501b483abd3b66bf98e6cef4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:58:21.469463 10987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431a014c633d4712979a338065b98fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:58:21.469478 10853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 431a014c633d4712979a338065b98fdb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:58:21.471998 11067 tablet_bootstrap.cc:492] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0: Bootstrap starting.
I20260924 02:58:21.472206 11068 tablet_bootstrap.cc:492] T 431a014c633d4712979a338065b98fdb P d47e43eedcf5407b9b3dfc0cfd77de7c: Bootstrap starting.
I20260924 02:58:21.472656 11067 tablet_bootstrap.cc:654] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0: Neither blocks nor log segments found. Creating new log.
I20260924 02:58:21.472796 11069 tablet_bootstrap.cc:492] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb: Bootstrap starting.
I20260924 02:58:21.473076 11068 tablet_bootstrap.cc:654] T 431a014c633d4712979a338065b98fdb P d47e43eedcf5407b9b3dfc0cfd77de7c: Neither blocks nor log segments found. Creating new log.
I20260924 02:58:21.473176 11067 log.cc:824] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0: Log is configured to *not* fsync() on all Append() calls
I20260924 02:58:21.473333 11069 tablet_bootstrap.cc:654] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb: Neither blocks nor log segments found. Creating new log.
I20260924 02:58:21.473382 11068 log.cc:824] T 431a014c633d4712979a338065b98fdb P d47e43eedcf5407b9b3dfc0cfd77de7c: Log is configured to *not* fsync() on all Append() calls
I20260924 02:58:21.473802 11069 log.cc:824] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb: Log is configured to *not* fsync() on all Append() calls
I20260924 02:58:21.474150 11067 tablet_bootstrap.cc:492] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0: No bootstrap required, opened a new log
I20260924 02:58:21.474208 11067 ts_tablet_manager.cc:1397] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:58:21.474434 11068 tablet_bootstrap.cc:492] T 431a014c633d4712979a338065b98fdb P d47e43eedcf5407b9b3dfc0cfd77de7c: No bootstrap required, opened a new log
I20260924 02:58:21.474490 11068 ts_tablet_manager.cc:1397] T 431a014c633d4712979a338065b98fdb P d47e43eedcf5407b9b3dfc0cfd77de7c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:58:21.474601 11069 tablet_bootstrap.cc:492] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb: No bootstrap required, opened a new log
I20260924 02:58:21.474665 11069 ts_tablet_manager.cc:1397] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 02:58:21.475805 11068 raft_consensus.cc:348] T 431a014c633d4712979a338065b98fdb P d47e43eedcf5407b9b3dfc0cfd77de7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e074a71a3f3a412c906e226df177b4fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44885 } } peers { permanent_uuid: "d47e43eedcf5407b9b3dfc0cfd77de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37791 } } peers { permanent_uuid: "045852134eb14c4186a96dd680cd88b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44199 } }
I20260924 02:58:21.475942 11068 raft_consensus.cc:374] T 431a014c633d4712979a338065b98fdb P d47e43eedcf5407b9b3dfc0cfd77de7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:58:21.475973 11068 raft_consensus.cc:729] T 431a014c633d4712979a338065b98fdb P d47e43eedcf5407b9b3dfc0cfd77de7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d47e43eedcf5407b9b3dfc0cfd77de7c, State: Initialized, Role: FOLLOWER
I20260924 02:58:21.476086 11068 consensus_queue.cc:260] T 431a014c633d4712979a338065b98fdb P d47e43eedcf5407b9b3dfc0cfd77de7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e074a71a3f3a412c906e226df177b4fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44885 } } peers { permanent_uuid: "d47e43eedcf5407b9b3dfc0cfd77de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37791 } } peers { permanent_uuid: "045852134eb14c4186a96dd680cd88b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44199 } }
I20260924 02:58:21.476073 11067 raft_consensus.cc:348] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e074a71a3f3a412c906e226df177b4fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44885 } } peers { permanent_uuid: "d47e43eedcf5407b9b3dfc0cfd77de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37791 } } peers { permanent_uuid: "045852134eb14c4186a96dd680cd88b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44199 } }
I20260924 02:58:21.476257 11067 raft_consensus.cc:374] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:58:21.476295 11067 raft_consensus.cc:729] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 045852134eb14c4186a96dd680cd88b0, State: Initialized, Role: FOLLOWER
I20260924 02:58:21.476357 11068 ts_tablet_manager.cc:1428] T 431a014c633d4712979a338065b98fdb P d47e43eedcf5407b9b3dfc0cfd77de7c: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 02:58:21.476424 11067 consensus_queue.cc:260] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e074a71a3f3a412c906e226df177b4fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44885 } } peers { permanent_uuid: "d47e43eedcf5407b9b3dfc0cfd77de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37791 } } peers { permanent_uuid: "045852134eb14c4186a96dd680cd88b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44199 } }
I20260924 02:58:21.476467 10785 heartbeater.cc:503] Master 127.0.98.62:34085 was elected leader, sending a full tablet report...
I20260924 02:58:21.476466 11069 raft_consensus.cc:348] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e074a71a3f3a412c906e226df177b4fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44885 } } peers { permanent_uuid: "d47e43eedcf5407b9b3dfc0cfd77de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37791 } } peers { permanent_uuid: "045852134eb14c4186a96dd680cd88b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44199 } }
I20260924 02:58:21.476635 11069 raft_consensus.cc:374] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:58:21.476682 11069 raft_consensus.cc:729] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e074a71a3f3a412c906e226df177b4fb, State: Initialized, Role: FOLLOWER
I20260924 02:58:21.476686 11067 ts_tablet_manager.cc:1428] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 02:58:21.476781 11069 consensus_queue.cc:260] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e074a71a3f3a412c906e226df177b4fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44885 } } peers { permanent_uuid: "d47e43eedcf5407b9b3dfc0cfd77de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37791 } } peers { permanent_uuid: "045852134eb14c4186a96dd680cd88b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44199 } }
I20260924 02:58:21.476818 11053 heartbeater.cc:503] Master 127.0.98.62:34085 was elected leader, sending a full tablet report...
I20260924 02:58:21.477016 11069 ts_tablet_manager.cc:1428] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 02:58:21.477133 10919 heartbeater.cc:503] Master 127.0.98.62:34085 was elected leader, sending a full tablet report...
W20260924 02:58:21.519974 10920 tablet.cc:2369] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 02:58:21.614674 10786 tablet.cc:2369] T 431a014c633d4712979a338065b98fdb P d47e43eedcf5407b9b3dfc0cfd77de7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 02:58:21.686918 11075 raft_consensus.cc:482] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:58:21.687054 11075 raft_consensus.cc:504] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e074a71a3f3a412c906e226df177b4fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44885 } } peers { permanent_uuid: "d47e43eedcf5407b9b3dfc0cfd77de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37791 } } peers { permanent_uuid: "045852134eb14c4186a96dd680cd88b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44199 } }
I20260924 02:58:21.687433 11075 leader_election.cc:288] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d47e43eedcf5407b9b3dfc0cfd77de7c (127.0.98.1:37791), 045852134eb14c4186a96dd680cd88b0 (127.0.98.3:44199)
I20260924 02:58:21.691078 11065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.687691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:57063 (local address 127.0.98.3:44199)
0924 02:58:21.687737 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:21.687739 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:21.687749 (+ 10us) server_negotiation.cc:407] Connection header received
0924 02:58:21.687778 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:21.687779 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:21.687791 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:21.687846 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:58:21.688001 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.688737 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.690160 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.690755 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.690820 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.690828 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:21.690963 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 02:58:21.691004 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 02:58:21.691109 11076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.687623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37791 (local address 127.0.98.2:44799)
0924 02:58:21.687799 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:21.687807 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:21.687858 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:21.687958 (+ 100us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:21.687959 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:21.687960 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:58:21.688017 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:21.688018 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.688828 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.688829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:21.690064 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:21.690067 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.690868 (+ 801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.690868 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:21.690940 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.690944 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:58:21.690993 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 02:58:21.691030 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":36,"thread_start_us":121,"threads_started":1}
I20260924 02:58:21.691123 11064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.687645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:44799 (local address 127.0.98.1:37791)
0924 02:58:21.687687 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:21.687689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:21.687864 (+ 175us) server_negotiation.cc:407] Connection header received
0924 02:58:21.687897 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:21.687898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:21.687909 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:21.687947 (+ 38us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:58:21.688072 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.688768 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.690169 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.690754 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.690800 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.690811 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:21.691019 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 02:58:21.691060 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 02:58:21.691147 11054 tablet.cc:2369] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 02:58:21.691356 10921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.687588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44199 (local address 127.0.98.2:57063)
0924 02:58:21.687643 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:21.687655 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:58:21.687697 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:58:21.687862 (+ 165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:58:21.687863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:58:21.687865 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:58:21.687933 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:21.687935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.688821 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.688822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:21.690066 (+ 1244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:58:21.690067 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.690822 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.690823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:58:21.690894 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.690898 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:58:21.690941 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 02:58:21.690972 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38,"mutex_wait_us":32}
I20260924 02:58:21.691563 10739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431a014c633d4712979a338065b98fdb" candidate_uuid: "e074a71a3f3a412c906e226df177b4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d47e43eedcf5407b9b3dfc0cfd77de7c" is_pre_election: true
I20260924 02:58:21.691627 11007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431a014c633d4712979a338065b98fdb" candidate_uuid: "e074a71a3f3a412c906e226df177b4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045852134eb14c4186a96dd680cd88b0" is_pre_election: true
I20260924 02:58:21.691700 10739 raft_consensus.cc:2454] T 431a014c633d4712979a338065b98fdb P d47e43eedcf5407b9b3dfc0cfd77de7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e074a71a3f3a412c906e226df177b4fb in term 0.
I20260924 02:58:21.691757 11007 raft_consensus.cc:2454] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e074a71a3f3a412c906e226df177b4fb in term 0.
I20260924 02:58:21.691923 10808 leader_election.cc:302] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d47e43eedcf5407b9b3dfc0cfd77de7c, e074a71a3f3a412c906e226df177b4fb; no voters:
I20260924 02:58:21.692058 11075 raft_consensus.cc:2781] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:58:21.692107 11075 raft_consensus.cc:482] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:58:21.692128 11075 raft_consensus.cc:3037] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:58:21.692770 11075 raft_consensus.cc:504] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e074a71a3f3a412c906e226df177b4fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44885 } } peers { permanent_uuid: "d47e43eedcf5407b9b3dfc0cfd77de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37791 } } peers { permanent_uuid: "045852134eb14c4186a96dd680cd88b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44199 } }
I20260924 02:58:21.692893 11075 leader_election.cc:288] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [CANDIDATE]: Term 1 election: Requested vote from peers d47e43eedcf5407b9b3dfc0cfd77de7c (127.0.98.1:37791), 045852134eb14c4186a96dd680cd88b0 (127.0.98.3:44199)
I20260924 02:58:21.693076 10739 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431a014c633d4712979a338065b98fdb" candidate_uuid: "e074a71a3f3a412c906e226df177b4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d47e43eedcf5407b9b3dfc0cfd77de7c"
I20260924 02:58:21.693073 11007 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "431a014c633d4712979a338065b98fdb" candidate_uuid: "e074a71a3f3a412c906e226df177b4fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "045852134eb14c4186a96dd680cd88b0"
I20260924 02:58:21.693141 10739 raft_consensus.cc:3037] T 431a014c633d4712979a338065b98fdb P d47e43eedcf5407b9b3dfc0cfd77de7c [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:58:21.693145 11007 raft_consensus.cc:3037] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:58:21.693780 10739 raft_consensus.cc:2454] T 431a014c633d4712979a338065b98fdb P d47e43eedcf5407b9b3dfc0cfd77de7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e074a71a3f3a412c906e226df177b4fb in term 1.
I20260924 02:58:21.693797 11007 raft_consensus.cc:2454] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e074a71a3f3a412c906e226df177b4fb in term 1.
I20260924 02:58:21.693933 10806 leader_election.cc:302] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 045852134eb14c4186a96dd680cd88b0, e074a71a3f3a412c906e226df177b4fb; no voters:
I20260924 02:58:21.694064 11075 raft_consensus.cc:2781] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:58:21.694178 11075 raft_consensus.cc:686] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [term 1 LEADER]: Becoming Leader. State: Replica: e074a71a3f3a412c906e226df177b4fb, State: Running, Role: LEADER
I20260924 02:58:21.694250 11075 consensus_queue.cc:237] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e074a71a3f3a412c906e226df177b4fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44885 } } peers { permanent_uuid: "d47e43eedcf5407b9b3dfc0cfd77de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37791 } } peers { permanent_uuid: "045852134eb14c4186a96dd680cd88b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44199 } }
I20260924 02:58:21.695007 10594 catalog_manager.cc:5495] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb reported cstate change: term changed from 0 to 1, leader changed from <none> to e074a71a3f3a412c906e226df177b4fb (127.0.98.2). New cstate: current_term: 1 leader_uuid: "e074a71a3f3a412c906e226df177b4fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e074a71a3f3a412c906e226df177b4fb" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 44885 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d47e43eedcf5407b9b3dfc0cfd77de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37791 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "045852134eb14c4186a96dd680cd88b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44199 } health_report { overall_health: UNKNOWN } } }
I20260924 02:58:21.750463 11066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.748205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60752 (local address 127.0.98.2:44885)
0924 02:58:21.748264 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 02:58:21.748266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:58:21.748284 (+ 18us) server_negotiation.cc:407] Connection header received
0924 02:58:21.748322 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:58:21.748322 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:58:21.748343 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:58:21.748399 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:58:21.748530 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.749010 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.749746 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:58:21.749933 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:58:21.749985 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:58:21.750145 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:58:21.750248 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:58:21.750316 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 02:58:21.750354 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 02:58:21.750374 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 02:58:21.752039 10739 raft_consensus.cc:1261] T 431a014c633d4712979a338065b98fdb P d47e43eedcf5407b9b3dfc0cfd77de7c [term 1 FOLLOWER]: Refusing update from remote peer e074a71a3f3a412c906e226df177b4fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:58:21.752038 11007 raft_consensus.cc:1261] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0 [term 1 FOLLOWER]: Refusing update from remote peer e074a71a3f3a412c906e226df177b4fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:58:21.752360 11077 consensus_queue.cc:1035] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "045852134eb14c4186a96dd680cd88b0" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:58:21.752485 11075 consensus_queue.cc:1035] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "d47e43eedcf5407b9b3dfc0cfd77de7c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:58:21.756995 10594 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:58:21.757102 10594 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60560:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:58:21.757792 10594 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bdee9f0fd7674ec79fc27f9eec787387: Sending DeleteTablet for 3 replicas of tablet 431a014c633d4712979a338065b98fdb
I20260924 02:58:21.758098 10853 tablet_service.cc:1514] Processing DeleteTablet for tablet 431a014c633d4712979a338065b98fdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:58:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60736
I20260924 02:58:21.758132 10987 tablet_service.cc:1514] Processing DeleteTablet for tablet 431a014c633d4712979a338065b98fdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:58:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:44566
I20260924 02:58:21.758301 11090 tablet_replica.cc:330] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb: stopping tablet replica
I20260924 02:58:21.758390 11090 raft_consensus.cc:2229] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [term 1 LEADER]: Raft consensus shutting down.
I20260924 02:58:21.758392 10719 tablet_service.cc:1514] Processing DeleteTablet for tablet 431a014c633d4712979a338065b98fdb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:58:21 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:34128
I20260924 02:58:21.758435 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 10656
I20260924 02:58:21.758498 11090 raft_consensus.cc:2258] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:58:21.759514 11090 ts_tablet_manager.cc:1905] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:58:21.761049 11091 tablet_replica.cc:330] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0: stopping tablet replica
I20260924 02:58:21.761160 11091 raft_consensus.cc:2229] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:58:21.761256 11091 raft_consensus.cc:2258] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:58:21.761255 11090 ts_tablet_manager.cc:1918] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:58:21.761315 11090 log.cc:1196] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/431a014c633d4712979a338065b98fdb
I20260924 02:58:21.761533 11090 ts_tablet_manager.cc:1939] T 431a014c633d4712979a338065b98fdb P e074a71a3f3a412c906e226df177b4fb: Deleting consensus metadata
I20260924 02:58:21.761675 11091 ts_tablet_manager.cc:1905] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:58:21.763159 11091 ts_tablet_manager.cc:1918] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:58:21.763216 11091 log.cc:1196] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/431a014c633d4712979a338065b98fdb
I20260924 02:58:21.763422 10579 catalog_manager.cc:4841] TS e074a71a3f3a412c906e226df177b4fb (127.0.98.2:44885): tablet 431a014c633d4712979a338065b98fdb (table test-table [id=6f5fdca4501b483abd3b66bf98e6cef4]) successfully deleted
I20260924 02:58:21.763453 11091 ts_tablet_manager.cc:1939] T 431a014c633d4712979a338065b98fdb P 045852134eb14c4186a96dd680cd88b0: Deleting consensus metadata
W20260924 02:58:21.763695 10580 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37791 (error 108)
I20260924 02:58:21.763804 10578 catalog_manager.cc:4841] TS 045852134eb14c4186a96dd680cd88b0 (127.0.98.3:44199): tablet 431a014c633d4712979a338065b98fdb (table test-table [id=6f5fdca4501b483abd3b66bf98e6cef4]) successfully deleted
I20260924 02:58:21.764256 11061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:58:21.764140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37791 (local address 127.0.0.1:34142)
0924 02:58:21.764188 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 02:58:21.764223 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37791: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 02:58:21.764271 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 10790
W20260924 02:58:21.764396 10580 catalog_manager.cc:4584] TS d47e43eedcf5407b9b3dfc0cfd77de7c (127.0.98.1:37791): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 431a014c633d4712979a338065b98fdb: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37791: connect: Connection refused (error 111)
I20260924 02:58:21.768464 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 10924
I20260924 02:58:21.772725 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 10562
2026-09-24T02:58:21Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2014 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:58:21 dist-test-slave-sb6p krb5kdc[11095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:58:21 dist-test-slave-sb6p krb5kdc[11095](info): set up 2 sockets
Sep 24 02:58:21 dist-test-slave-sb6p krb5kdc[11095](info): commencing operation
krb5kdc: starting...
W20260924 02:58:22.751078 392 mini_kdc.cc:121] Time spent starting KDC: real 0.959s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:58:22 dist-test-slave-sb6p krb5kdc[11095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218702, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:58:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:58:22Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 02:58:22.898160 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPoI4uN/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/postgres -l logfile start
2026-09-24 02:58:25.188 UTC [11125] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 02:58:25.188 UTC [11125] LOG: listening on IPv4 address "127.0.98.20", port 53047
2026-09-24 02:58:25.191 UTC [11125] LOG: listening on Unix socket "/tmp/.s.PGSQL.53047"
2026-09-24 02:58:25.217 UTC [11128] LOG: database system was shut down at 2026-09-24 02:58:23 UTC
2026-09-24 02:58:25.220 UTC [11125] LOG: database system is ready to accept connections
I20260924 02:58:26.099138 392 mini_postgres.cc:96] Postgres bound to 53047
2026-09-24 02:58:26.104 UTC [11136] FATAL: database "slave" does not exist
127.0.98.20:53047 - accepting connections
I20260924 02:58:26.105047 392 mini_ranger.cc:161] Starting Ranger...
I20260924 02:58:26.118701 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 02:58:26.553259 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 02:58:26.553330 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ranger-admin
2026-09-24 02:58:26,809 [I] DB FLAVOR :POSTGRES
2026-09-24 02:58:26,810 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 02:58:26,810 [I] Checking connection..
2026-09-24 02:58:26,810 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 02:58:27,060 [I] Checking connection passed.
2026-09-24 02:58:27,061 [I] --------- Verifying version history table ---------
2026-09-24 02:58:27,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 02:58:27,304 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 02:58:27,304 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 02:58:27,560 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 02:58:27,560 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 02:58:27,560 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 02:58:27.803 UTC [11200] WARNING: there is no transaction in progress
2026-09-24 02:58:27,809 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 02:58:27,809 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 02:58:28,039 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 02:58:28,039 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 02:58:28,039 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 02:58:28,291 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 02:58:28,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-sb6p.c.gcp-upstream.internal','N') ;"
2026-09-24 02:58:28,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 02:58:29,023 [I] Table x_portal_user does not exist in database ranger
2026-09-24 02:58:29,023 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 02:58:29,249 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 02:58:29,249 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 02:58:29,491 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 02:58:29,725 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 02:58:29,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 02:58:30.621 UTC [11308] WARNING: there is no transaction in progress
2026-09-24 02:58:30.636 UTC [11308] WARNING: there is no transaction in progress
2026-09-24 02:58:30.651 UTC [11308] WARNING: there is no transaction in progress
2026-09-24 02:58:30.666 UTC [11308] WARNING: there is no transaction in progress
2026-09-24 02:58:30.681 UTC [11308] WARNING: there is no transaction in progress
2026-09-24 02:58:30.698 UTC [11308] WARNING: there is no transaction in progress
2026-09-24 02:58:30.761 UTC [11308] WARNING: there is no transaction in progress
2026-09-24 02:58:30.767 UTC [11308] WARNING: there is no transaction in progress
2026-09-24 02:58:30.782 UTC [11308] WARNING: there is no transaction in progress
2026-09-24 02:58:30.792 UTC [11308] WARNING: there is no transaction in progress
2026-09-24 02:58:30.802 UTC [11308] WARNING: there is no transaction in progress
2026-09-24 02:58:30.815 UTC [11308] WARNING: there is no transaction in progress
2026-09-24 02:58:30.824 UTC [11308] WARNING: there is no transaction in progress
2026-09-24 02:58:30.831 UTC [11308] WARNING: there is no transaction in progress
2026-09-24 02:58:30.838 UTC [11308] WARNING: there is no transaction in progress
2026-09-24 02:58:30.844 UTC [11308] WARNING: there is no transaction in progress
2026-09-24 02:58:31,447 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 02:58:31,448 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 02:58:31,692 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 02:58:31,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 02:58:31,926 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 02:58:31,926 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 02:58:32,183 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 02:58:32,184 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 02:58:32,423 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 02:58:32,665 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-sb6p.c.gcp-upstream.internal';"
2026-09-24 02:58:32,891 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 02:58:32,891 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 02:58:33,110 [I] DB_PATCHES have already been applied
I20260924 02:58:33.115906 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPoI4uN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 02:58:33.115967 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 02:58:33.118944 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:57341
Sep 24, 2026 2:58:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 2:58:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 2:58:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 2:58:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 2:58:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 2:58:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 2:58:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 02:58:34.130277 392 mini_ranger.cc:160] Time spent starting Ranger: real 8.025s user 0.001s sys 0.006s
Sep 24, 2026 2:58:53 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 2:58:53 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 24, 2026 2:58:53 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 2:58:53 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 2:58:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 2:58:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 2:58:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 2:58:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 2:58:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 2:58:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 2:58:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 2:58:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 2:58:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 2:58:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 2:58:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 2:58:53 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 2:58:53 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 2:58:54 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 02:58:54.659338 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 02:58:54 dist-test-slave-sb6p krb5kdc[11095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218734, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 02:58:54.747905 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 02:58:54.756800 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 02:58:55.525266 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 02:58:55.525348 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ranger-kms
I20260924 02:58:55.525363 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:53047
2026-09-24 02:58:55,684 [I] DB FLAVOR :POSTGRES
2026-09-24 02:58:55,684 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 02:58:55,684 [I] Checking connection
2026-09-24 02:58:55,684 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 02:58:55,915 [I] connection success
2026-09-24 02:58:55,916 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 02:58:55,916 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 02:58:55,916 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 02:58:56,150 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 02:58:56,150 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 02:58:56,150 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 02:58:56,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:53047/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 02:58:56,409 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 02:58:56.856513 392 mini_ranger_kms.cc:310] Created kms service
I20260924 02:58:57.058106 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 02:58:57.097265 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 02:58:57 dist-test-slave-sb6p krb5kdc[11095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218734, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 02:58:57.447283 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 02:58:57.447395 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 02:58:57.447417 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 02:58:57.450814 392 mini_ranger_kms.cc:276] Ranger KMS PID: 11488
I20260924 02:58:57.450907 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:56893
Sep 24, 2026 2:58:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 2:58:58 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 2:58:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 2:58:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 2:58:58 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 2:58:58 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 02:58:59.460809 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.713s user 0.011s sys 0.130s
I20260924 02:58:59.460917 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 02:58:59.460991 392 mini_ranger_kms.cc:401] 127.0.98.20:56893/kms/v1/keys
Sep 24 02:58:59 dist-test-slave-sb6p krb5kdc[11095](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790218739, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 02:58:59 dist-test-slave-sb6p krb5kdc[11095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218739, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 02:59:00 dist-test-slave-sb6p krb5kdc[11095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218739, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 02:59:02 dist-test-slave-sb6p krb5kdc[11095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 02:59:02.627761 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34995
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34995
--ranger_config_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56893/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:02.706756 11539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:02.706965 11539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:02.706986 11539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:02.708097 11539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:59:02.708132 11539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:02.708145 11539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:02.708156 11539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:59:02.708168 11539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:59:02.709621 11539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:56893/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34995
--ranger_config_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34995
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11539
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:02.709789 11539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:02.709951 11539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:02.714443 11544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:02.714444 11547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:02.714440 11545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:02.714648 11539 server_base.cc:1034] running on GCE node
I20260924 02:59:02.714838 11539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:02.715097 11539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:02.716224 11539 hybrid_clock.cc:648] HybridClock initialized: now 1790218742716204 us; error 38 us; skew 500 ppm
Sep 24 02:59:02 dist-test-slave-sb6p krb5kdc[11095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:02.721488 11539 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:59:02.721910 11539 webserver.cc:460] Webserver started at http://127.0.98.62:43679/ using document root <none> and password file <none>
I20260924 02:59:02.722090 11539 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:02.722123 11539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:02.722209 11539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 02:59:02 dist-test-slave-sb6p krb5kdc[11095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218742, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 02:59:02.771950 11539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "53893747257740b490798f7bb9602427"
format_stamp: "Formatted at 2026-09-24 02:59:02 on dist-test-slave-sb6p"
server_key: "3986a7fbe1951e58113bff858728787a"
server_key_iv: "81a4ee2b09f23691de22526d6a61e5a6"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:02.772475 11539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "53893747257740b490798f7bb9602427"
format_stamp: "Formatted at 2026-09-24 02:59:02 on dist-test-slave-sb6p"
server_key: "3986a7fbe1951e58113bff858728787a"
server_key_iv: "81a4ee2b09f23691de22526d6a61e5a6"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:02.774031 11539 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 02:59:02.792691 11554 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:02.793447 11539 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:59:02.793596 11539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "53893747257740b490798f7bb9602427"
format_stamp: "Formatted at 2026-09-24 02:59:02 on dist-test-slave-sb6p"
server_key: "3986a7fbe1951e58113bff858728787a"
server_key_iv: "81a4ee2b09f23691de22526d6a61e5a6"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:02.793725 11539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:02.807914 11539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:02.810101 11539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:02.810245 11539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:02.815248 11539 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34995
I20260924 02:59:02.815301 11606 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34995 every 8 connection(s)
I20260924 02:59:02.815768 11539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:59:02.816515 11607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:02.819537 11607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427: Bootstrap starting.
I20260924 02:59:02.820243 11607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:02.820549 11607 log.cc:824] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:02.821332 11607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427: No bootstrap required, opened a new log
I20260924 02:59:02.822835 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 11539
I20260924 02:59:02.822942 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:59:02.823138 392 external_mini_cluster.cc:1428] Setting key 13ac8dd1cbbf34723b11d5afad025250
I20260924 02:59:02.824560 11607 raft_consensus.cc:348] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53893747257740b490798f7bb9602427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34995 } }
I20260924 02:59:02.824729 11607 raft_consensus.cc:374] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:02.824762 11607 raft_consensus.cc:729] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53893747257740b490798f7bb9602427, State: Initialized, Role: FOLLOWER
I20260924 02:59:02.824883 11607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53893747257740b490798f7bb9602427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34995 } }
I20260924 02:59:02.824951 11607 raft_consensus.cc:388] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:59:02.824986 11607 raft_consensus.cc:482] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:59:02.825028 11607 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:02.825652 11607 raft_consensus.cc:504] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53893747257740b490798f7bb9602427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34995 } }
I20260924 02:59:02.825757 11607 leader_election.cc:302] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 53893747257740b490798f7bb9602427; no voters:
I20260924 02:59:02.825904 11607 leader_election.cc:288] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:59:02.825927 11612 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:59:02.826079 11612 raft_consensus.cc:686] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [term 1 LEADER]: Becoming Leader. State: Replica: 53893747257740b490798f7bb9602427, State: Running, Role: LEADER
I20260924 02:59:02.826145 11607 sys_catalog.cc:564] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:59:02.826205 11612 consensus_queue.cc:237] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53893747257740b490798f7bb9602427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34995 } }
I20260924 02:59:02.826562 11614 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "53893747257740b490798f7bb9602427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53893747257740b490798f7bb9602427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34995 } } }
I20260924 02:59:02.826639 11614 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [sys.catalog]: This master's current role is: LEADER
I20260924 02:59:02.826555 11612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 53893747257740b490798f7bb9602427. Latest consensus state: current_term: 1 leader_uuid: "53893747257740b490798f7bb9602427" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "53893747257740b490798f7bb9602427" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34995 } } }
I20260924 02:59:02.826844 11612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427 [sys.catalog]: This master's current role is: LEADER
I20260924 02:59:02.827621 11607 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 02:59:02.829972 11620 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:59:02.831035 11620 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 02:59:02 dist-test-slave-sb6p krb5kdc[11095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218742, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:02.839159 11620 catalog_manager.cc:1347] Generated new cluster ID: 05d681c3dc684a6cb206cc96d4271219
I20260924 02:59:02.839278 11620 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:59:02.840848 11611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:02.823598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40662 (local address 127.0.98.62:34995)
0924 02:59:02.829886 (+ 6288us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:02.829900 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:02.829963 (+ 63us) server_negotiation.cc:407] Connection header received
0924 02:59:02.830274 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:02.830282 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:02.830372 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:02.830583 (+ 211us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:02.831374 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:02.832834 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:02.833619 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:02.834061 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:02.834157 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:02.836391 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:02.836395 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:02.836395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:02.836405 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:02.838771 (+ 2366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:02.839157 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:02.839160 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:02.839161 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:02.839307 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:02.839520 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:02.839521 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:02.839521 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:02.839850 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:02.840405 (+ 555us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:02.840480 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 02:59:02.840615 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6143,"thread_start_us":71,"threads_started":1}
I20260924 02:59:02.877565 11620 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:59:02.878314 11620 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:59:02.881657 11620 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427: Generated new TSK 0
I20260924 02:59:02.881827 11620 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 02:59:03.520701 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34995
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:03.604090 11652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:03.604288 11652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:03.604315 11652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:03.605450 11652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:03.605491 11652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:03.605525 11652 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:59:03.606976 11652 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:56893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11652
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:03.607189 11652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:03.607384 11652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:03.612534 11660 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:03.612571 11657 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:03.612530 11658 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:03.612825 11652 server_base.cc:1034] running on GCE node
I20260924 02:59:03.613066 11652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:03.613277 11652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:03.614408 11652 hybrid_clock.cc:648] HybridClock initialized: now 1790218743614395 us; error 29 us; skew 500 ppm
Sep 24 02:59:03 dist-test-slave-sb6p krb5kdc[11095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:03.616871 11652 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 02:59:03.617300 11652 webserver.cc:460] Webserver started at http://127.0.98.1:39337/ using document root <none> and password file <none>
I20260924 02:59:03.617513 11652 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:03.617560 11652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:03.617682 11652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 02:59:03 dist-test-slave-sb6p krb5kdc[11095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 02:59:03.631062 11652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "fc3e1a95535049ffabe4b485a96061f6"
format_stamp: "Formatted at 2026-09-24 02:59:03 on dist-test-slave-sb6p"
server_key: "b1f776e236da1545eced3f5c8bde997c"
server_key_iv: "a0c0b04c2e5424f9c9611818d43a83db"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:03.631440 11652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fc3e1a95535049ffabe4b485a96061f6"
format_stamp: "Formatted at 2026-09-24 02:59:03 on dist-test-slave-sb6p"
server_key: "b1f776e236da1545eced3f5c8bde997c"
server_key_iv: "a0c0b04c2e5424f9c9611818d43a83db"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:03.632710 11652 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:59:03.645262 11667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:03.645483 11652 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:59:03.645576 11652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "fc3e1a95535049ffabe4b485a96061f6"
format_stamp: "Formatted at 2026-09-24 02:59:03 on dist-test-slave-sb6p"
server_key: "b1f776e236da1545eced3f5c8bde997c"
server_key_iv: "a0c0b04c2e5424f9c9611818d43a83db"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:03.645661 11652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:03.657394 11652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:03.659574 11652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:03.659767 11652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:03.660063 11652 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:03.660447 11652 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:59:03.660527 11652 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:03.660622 11652 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:59:03.660691 11652 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:03.667783 11652 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38831
I20260924 02:59:03.668258 11652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:59:03.668709 11780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38831 every 8 connection(s)
Sep 24 02:59:03 dist-test-slave-sb6p krb5kdc[11095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:03.678752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 11652
I20260924 02:59:03.678855 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:59:03.679061 392 external_mini_cluster.cc:1428] Setting key 9bdd5cc81cf03f6fc6c71576a1f4b356
I20260924 02:59:03.680505 11783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:03.669095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34995 (local address 127.0.98.1:36899)
0924 02:59:03.669238 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:03.669282 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:03.669881 (+ 599us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:03.670311 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:03.670315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:03.670583 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:03.670866 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:03.670873 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:03.671897 (+ 1024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:03.671898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:03.673075 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:03.673078 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:03.673695 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:03.673696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:03.673839 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:03.674415 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:03.674426 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:03.676791 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:03.678821 (+ 2030us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:03.678828 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:03.678830 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:03.679130 (+ 300us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:03.679425 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:03.679426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:03.679426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:03.679553 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:03.679831 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:03.679836 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:03.680011 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 02:59:03.680126 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 02:59:03.680275 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":79,"threads_started":1}
I20260924 02:59:03.681206 11784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:03.669386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:36899 (local address 127.0.98.62:34995)
0924 02:59:03.670079 (+ 693us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:03.670082 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:03.670090 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:59:03.670115 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:03.670116 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:03.670128 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:03.670192 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:03.670969 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:03.671796 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:03.673219 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:03.673561 (+ 342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:03.673629 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:03.676895 (+ 3266us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:03.676898 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:03.676898 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:03.676906 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:03.678727 (+ 1821us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:03.679293 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:03.679294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:03.679295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:03.679362 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:03.679620 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:03.679621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:03.679621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:03.679754 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:03.679808 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:03.681006 (+ 1198us) server_negotiation.cc:299] Negotiation successful
0924 02:59:03.681105 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":663,"thread_start_us":73,"threads_started":1}
I20260924 02:59:03.681733 11781 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34995
I20260924 02:59:03.681864 11781 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:03.682096 11781 heartbeater.cc:511] Master 127.0.98.62:34995 requested a full tablet report, sending...
I20260924 02:59:03.682641 11571 ts_manager.cc:194] Registered new tserver with Master: fc3e1a95535049ffabe4b485a96061f6 (127.0.98.1:38831)
I20260924 02:59:03.683511 11571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:36899
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 02:59:03.707150 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34995
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:03.835311 11789 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:03.835659 11789 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:03.835745 11789 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:03.837471 11789 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:03.837585 11789 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:03.837668 11789 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:59:03.839905 11789 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:56893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11789
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:03.840343 11789 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:03.840669 11789 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:03.847353 11799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:03.847930 11796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:03.848131 11797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:03.848361 11789 server_base.cc:1034] running on GCE node
I20260924 02:59:03.848652 11789 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:03.848932 11789 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:03.850103 11789 hybrid_clock.cc:648] HybridClock initialized: now 1790218743850084 us; error 35 us; skew 500 ppm
Sep 24 02:59:03 dist-test-slave-sb6p krb5kdc[11095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:03.853354 11789 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 02:59:03.853844 11789 webserver.cc:460] Webserver started at http://127.0.98.2:40405/ using document root <none> and password file <none>
I20260924 02:59:03.854085 11789 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:03.854135 11789 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:03.854251 11789 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 02:59:03 dist-test-slave-sb6p krb5kdc[11095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 02:59:03.871343 11789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "8718a7dc82624a97a3f7e4f7b5157e03"
format_stamp: "Formatted at 2026-09-24 02:59:03 on dist-test-slave-sb6p"
server_key: "11461bfe959c49f776e2a2d14d49ae8e"
server_key_iv: "fdadca61679bb1f31a6cec1f67e74666"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:03.871879 11789 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "8718a7dc82624a97a3f7e4f7b5157e03"
format_stamp: "Formatted at 2026-09-24 02:59:03 on dist-test-slave-sb6p"
server_key: "11461bfe959c49f776e2a2d14d49ae8e"
server_key_iv: "fdadca61679bb1f31a6cec1f67e74666"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:03.874423 11789 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
Sep 24 02:59:03 dist-test-slave-sb6p krb5kdc[11095](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790218743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:03.897495 11806 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:03.897745 11789 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260924 02:59:03.897845 11789 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "8718a7dc82624a97a3f7e4f7b5157e03"
format_stamp: "Formatted at 2026-09-24 02:59:03 on dist-test-slave-sb6p"
server_key: "11461bfe959c49f776e2a2d14d49ae8e"
server_key_iv: "fdadca61679bb1f31a6cec1f67e74666"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:03.897939 11789 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:03.915077 11789 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:03.917390 11789 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:03.917578 11789 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:03.917830 11789 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:03.918234 11789 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:59:03.918316 11789 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:03.918381 11789 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:59:03.918433 11789 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:03.924835 11789 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43233
I20260924 02:59:03.925312 11789 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 02:59:03.928303 11919 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43233 every 8 connection(s)
I20260924 02:59:03.933629 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 11789
I20260924 02:59:03.933735 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:59:03.933900 392 external_mini_cluster.cc:1428] Setting key 3b6c31d4bfb663dd5cc888fb676384a4
Sep 24 02:59:03 dist-test-slave-sb6p krb5kdc[11095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218743, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:03.939456 11784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:03.929248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:50615 (local address 127.0.98.62:34995)
0924 02:59:03.929292 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:03.929294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:03.929768 (+ 474us) server_negotiation.cc:407] Connection header received
0924 02:59:03.930432 (+ 664us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:03.930433 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:03.930444 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:03.930472 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:03.931289 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:03.932056 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:03.933517 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:03.933797 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:03.933838 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:03.936393 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:03.936395 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:03.936395 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:03.936402 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:03.937927 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:03.938392 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:03.938394 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:03.938394 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:03.938448 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:03.938721 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:03.938722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:03.938722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:03.938858 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:03.938897 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:03.939277 (+ 380us) server_negotiation.cc:299] Negotiation successful
0924 02:59:03.939361 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 02:59:03.940186 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:03.928974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34995 (local address 127.0.98.2:50615)
0924 02:59:03.929534 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:03.929687 (+ 153us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:03.930318 (+ 631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:03.930677 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:03.930681 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:03.930911 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:03.931186 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:03.931192 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:03.932164 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:03.932164 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:03.933420 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:03.933424 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:03.934017 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:03.934018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:03.934177 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:03.934783 (+ 606us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:03.934794 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:03.936293 (+ 1499us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:03.938018 (+ 1725us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:03.938024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:03.938026 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:03.938310 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:03.938511 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:03.938511 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:03.938511 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:03.938650 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:03.938940 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:03.938945 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:03.939130 (+ 185us) client_negotiation.cc:769] Sending connection context
0924 02:59:03.939777 (+ 647us) client_negotiation.cc:240] Negotiation successful
0924 02:59:03.939939 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":83,"threads_started":1}
I20260924 02:59:03.940794 11920 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34995
I20260924 02:59:03.940882 11920 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:03.941067 11920 heartbeater.cc:511] Master 127.0.98.62:34995 requested a full tablet report, sending...
I20260924 02:59:03.941453 11571 ts_manager.cc:194] Registered new tserver with Master: 8718a7dc82624a97a3f7e4f7b5157e03 (127.0.98.2:43233)
I20260924 02:59:03.941941 11571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:50615
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 02:59:03.967711 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:56893/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34995
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35905
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:04.080968 11925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:04.081324 11925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:04.081425 11925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:04.083282 11925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:04.083408 11925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:04.083489 11925 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:59:04.085759 11925 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35905
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:56893/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:34995
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:04.086289 11925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:04.086678 11925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:04.093482 11932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:04.093562 11934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:04.093482 11931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:04.095155 11925 server_base.cc:1034] running on GCE node
I20260924 02:59:04.095377 11925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:04.095624 11925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:04.102936 11925 hybrid_clock.cc:648] HybridClock initialized: now 1790218744102905 us; error 40 us; skew 500 ppm
Sep 24 02:59:04 dist-test-slave-sb6p krb5kdc[11095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:04.106276 11925 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 02:59:04.106846 11925 webserver.cc:460] Webserver started at http://127.0.98.3:44461/ using document root <none> and password file <none>
I20260924 02:59:04.107155 11925 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:04.107254 11925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:04.107412 11925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 02:59:04 dist-test-slave-sb6p krb5kdc[11095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 02:59:04.127874 11925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "4f3c46f0b3f341d6aa0350937774a75a"
format_stamp: "Formatted at 2026-09-24 02:59:04 on dist-test-slave-sb6p"
server_key: "9a20f14cb8862d5443e371b11f0e8fdc"
server_key_iv: "ceb53194a30e2dd05021bd40fb9367a9"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:04.129623 11925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4f3c46f0b3f341d6aa0350937774a75a"
format_stamp: "Formatted at 2026-09-24 02:59:04 on dist-test-slave-sb6p"
server_key: "9a20f14cb8862d5443e371b11f0e8fdc"
server_key_iv: "ceb53194a30e2dd05021bd40fb9367a9"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:04.134919 11925 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.000s
I20260924 02:59:04.151887 11941 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:04.152148 11925 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260924 02:59:04.152244 11925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "4f3c46f0b3f341d6aa0350937774a75a"
format_stamp: "Formatted at 2026-09-24 02:59:04 on dist-test-slave-sb6p"
server_key: "9a20f14cb8862d5443e371b11f0e8fdc"
server_key_iv: "ceb53194a30e2dd05021bd40fb9367a9"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:04.152347 11925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:04.170815 11925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:04.173084 11925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:04.173265 11925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:04.173507 11925 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:04.174029 11925 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:59:04.174103 11925 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:04.174154 11925 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:59:04.174190 11925 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:04.181483 11925 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:42765
I20260924 02:59:04.181955 11925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 02:59:04.189123 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 11925
I20260924 02:59:04.189234 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:59:04.189396 392 external_mini_cluster.cc:1428] Setting key b00adb6692ac077e69c95b9b3524a5f6
Sep 24 02:59:04 dist-test-slave-sb6p krb5kdc[11095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218744, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:04.194924 12054 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:42765 every 8 connection(s)
I20260924 02:59:04.195358 11784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.183929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:41815 (local address 127.0.98.62:34995)
0924 02:59:04.183977 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:04.183979 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:04.184186 (+ 207us) server_negotiation.cc:407] Connection header received
0924 02:59:04.184866 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:04.184867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:04.184880 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:04.184909 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:04.185744 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.186484 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.187890 (+ 1406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.188161 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.188199 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.192028 (+ 3829us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:04.192029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:04.192030 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:04.192037 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:04.193670 (+ 1633us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:04.194172 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:04.194173 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:04.194174 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:04.194230 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:04.194505 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:04.194506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:04.194507 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:04.194664 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:04.194704 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:04.195178 (+ 474us) server_negotiation.cc:299] Negotiation successful
0924 02:59:04.195264 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 02:59:04.196085 12057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.183646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34995 (local address 127.0.98.3:41815)
0924 02:59:04.184058 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:04.184104 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:04.184745 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:04.185124 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:04.185128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:04.185357 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:04.185635 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:04.185641 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.186627 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.186628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:04.187798 (+ 1170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:04.187801 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.188281 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.188282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:04.188411 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.190242 (+ 1831us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:04.190253 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:04.191922 (+ 1669us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:04.193768 (+ 1846us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:04.193774 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:04.193777 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:04.194072 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:04.194295 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:04.194296 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:04.194296 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:04.194435 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:04.194751 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:04.194757 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:04.194950 (+ 193us) client_negotiation.cc:769] Sending connection context
0924 02:59:04.195694 (+ 744us) client_negotiation.cc:240] Negotiation successful
0924 02:59:04.195862 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":82,"threads_started":1}
I20260924 02:59:04.196702 12055 heartbeater.cc:347] Connected to a master server at 127.0.98.62:34995
I20260924 02:59:04.196792 12055 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:04.196972 12055 heartbeater.cc:511] Master 127.0.98.62:34995 requested a full tablet report, sending...
I20260924 02:59:04.197388 11571 ts_manager.cc:194] Registered new tserver with Master: 4f3c46f0b3f341d6aa0350937774a75a (127.0.98.3:42765)
I20260924 02:59:04.197893 11571 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:41815
I20260924 02:59:04.200840 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 02:59:04.214380 11784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.206795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52064 (local address 127.0.98.62:34995)
0924 02:59:04.206836 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:04.206838 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:04.207738 (+ 900us) server_negotiation.cc:407] Connection header received
0924 02:59:04.207844 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:04.207845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:04.207856 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:04.207885 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:04.208662 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.209459 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.210648 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.210951 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.210992 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.212201 (+ 1209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:04.212203 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:04.212203 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:04.212210 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:04.213411 (+ 1201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:04.213711 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:04.213712 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:04.213713 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:04.213760 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:04.213917 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:04.213918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:04.213918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:04.214022 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:04.214108 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:04.214208 (+ 100us) server_negotiation.cc:299] Negotiation successful
0924 02:59:04.214282 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 02:59:04.215981 11571 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52064:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 02:59:04.217617 11571 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:59:04.224309 12068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.219947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42765 (local address 127.0.0.1:46858)
0924 02:59:04.220322 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:04.220332 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:04.220389 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:04.220644 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:04.220647 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:04.220649 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:04.220710 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:04.220712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.221887 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.221888 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:04.223059 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:04.223061 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.224026 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.224027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:04.224104 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.224117 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 02:59:04.224180 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 02:59:04.224229 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":360,"mutex_wait_us":4,"thread_start_us":68,"threads_started":1}
I20260924 02:59:04.224324 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.220245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39822 (local address 127.0.98.1:38831)
0924 02:59:04.220622 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:04.220626 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:04.220635 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:59:04.220661 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:04.220662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:04.220675 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:04.220736 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:04.221026 (+ 290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.221689 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.223159 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.223873 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.223917 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.223971 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:04.224213 (+ 242us) server_negotiation.cc:299] Negotiation successful
0924 02:59:04.224254 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":47,"threads_started":1}
I20260924 02:59:04.224498 12067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.219920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38831 (local address 127.0.0.1:39822)
0924 02:59:04.220051 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:04.220090 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:04.220149 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:04.220745 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:04.220747 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:04.220748 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:04.220798 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:04.220800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.221765 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.221766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:04.223074 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:04.223077 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.224037 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.224038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:04.224115 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.224118 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:59:04.224391 (+ 273us) client_negotiation.cc:240] Negotiation successful
0924 02:59:04.224430 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":1,"thread_start_us":69,"threads_started":1}
I20260924 02:59:04.224995 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.220170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46858 (local address 127.0.98.3:42765)
0924 02:59:04.220326 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:04.220330 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:04.220380 (+ 50us) server_negotiation.cc:407] Connection header received
0924 02:59:04.220472 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:04.220474 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:04.220490 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:04.220550 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:04.220774 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.221800 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.223164 (+ 1364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.223863 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.223926 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.223994 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:04.224879 (+ 885us) server_negotiation.cc:299] Negotiation successful
0924 02:59:04.224920 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":74,"threads_started":1}
I20260924 02:59:04.225430 11715 tablet_service.cc:1467] Processing CreateTablet for tablet e1a13d78456642e986bdcaefdefdc78d (DEFAULT_TABLE table=test-table [id=1f992eb165e642c4849f799acdaa50b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:59:04.225790 11715 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1a13d78456642e986bdcaefdefdc78d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:04.226060 12069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.220122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43233 (local address 127.0.0.1:51000)
0924 02:59:04.220463 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:04.220472 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:04.220530 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:04.221981 (+ 1451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:04.221981 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:04.221983 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:04.222056 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:04.222059 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.223580 (+ 1521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.223581 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:04.224599 (+ 1018us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:04.224601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.225866 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.225867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:04.225926 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.225930 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:59:04.225959 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 02:59:04.225994 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"mutex_wait_us":24,"thread_start_us":60,"threads_started":1}
I20260924 02:59:04.225955 11989 tablet_service.cc:1467] Processing CreateTablet for tablet e1a13d78456642e986bdcaefdefdc78d (DEFAULT_TABLE table=test-table [id=1f992eb165e642c4849f799acdaa50b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:59:04.226271 11989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1a13d78456642e986bdcaefdefdc78d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:04.226409 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.220443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51000 (local address 127.0.98.2:43233)
0924 02:59:04.221829 (+ 1386us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:04.221834 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:04.221842 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:59:04.221876 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:04.221878 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:04.221893 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:04.222002 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:04.222173 (+ 171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.223484 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.224730 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.225770 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.226198 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.226278 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:04.226303 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 02:59:04.226339 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1351,"thread_start_us":75,"threads_started":1}
I20260924 02:59:04.227001 11854 tablet_service.cc:1467] Processing CreateTablet for tablet e1a13d78456642e986bdcaefdefdc78d (DEFAULT_TABLE table=test-table [id=1f992eb165e642c4849f799acdaa50b0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:59:04.227301 11854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1a13d78456642e986bdcaefdefdc78d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:04.229547 12074 tablet_bootstrap.cc:492] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a: Bootstrap starting.
I20260924 02:59:04.230144 12075 tablet_bootstrap.cc:492] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03: Bootstrap starting.
I20260924 02:59:04.230614 12074 tablet_bootstrap.cc:654] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:04.230693 12073 tablet_bootstrap.cc:492] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6: Bootstrap starting.
I20260924 02:59:04.231043 12074 log.cc:824] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:04.231114 12075 tablet_bootstrap.cc:654] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:04.231353 12073 tablet_bootstrap.cc:654] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:04.231416 12075 log.cc:824] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:04.231717 12073 log.cc:824] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:04.231817 12074 tablet_bootstrap.cc:492] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a: No bootstrap required, opened a new log
I20260924 02:59:04.231885 12074 ts_tablet_manager.cc:1397] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.001s
I20260924 02:59:04.233808 12074 raft_consensus.cc:348] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3e1a95535049ffabe4b485a96061f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38831 } } peers { permanent_uuid: "8718a7dc82624a97a3f7e4f7b5157e03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43233 } } peers { permanent_uuid: "4f3c46f0b3f341d6aa0350937774a75a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42765 } }
I20260924 02:59:04.234020 12074 raft_consensus.cc:374] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:04.234061 12074 raft_consensus.cc:729] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f3c46f0b3f341d6aa0350937774a75a, State: Initialized, Role: FOLLOWER
I20260924 02:59:04.234191 12074 consensus_queue.cc:260] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3e1a95535049ffabe4b485a96061f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38831 } } peers { permanent_uuid: "8718a7dc82624a97a3f7e4f7b5157e03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43233 } } peers { permanent_uuid: "4f3c46f0b3f341d6aa0350937774a75a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42765 } }
I20260924 02:59:04.234452 12074 ts_tablet_manager.cc:1428] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 02:59:04.234462 12055 heartbeater.cc:503] Master 127.0.98.62:34995 was elected leader, sending a full tablet report...
I20260924 02:59:04.235147 12073 tablet_bootstrap.cc:492] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6: No bootstrap required, opened a new log
I20260924 02:59:04.235211 12073 ts_tablet_manager.cc:1397] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.000s
I20260924 02:59:04.235265 12075 tablet_bootstrap.cc:492] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03: No bootstrap required, opened a new log
I20260924 02:59:04.235313 12075 ts_tablet_manager.cc:1397] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260924 02:59:04.237087 12075 raft_consensus.cc:348] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3e1a95535049ffabe4b485a96061f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38831 } } peers { permanent_uuid: "8718a7dc82624a97a3f7e4f7b5157e03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43233 } } peers { permanent_uuid: "4f3c46f0b3f341d6aa0350937774a75a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42765 } }
I20260924 02:59:04.237232 12075 raft_consensus.cc:374] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:04.237262 12075 raft_consensus.cc:729] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8718a7dc82624a97a3f7e4f7b5157e03, State: Initialized, Role: FOLLOWER
I20260924 02:59:04.237372 12075 consensus_queue.cc:260] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3e1a95535049ffabe4b485a96061f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38831 } } peers { permanent_uuid: "8718a7dc82624a97a3f7e4f7b5157e03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43233 } } peers { permanent_uuid: "4f3c46f0b3f341d6aa0350937774a75a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42765 } }
I20260924 02:59:04.237620 12075 ts_tablet_manager.cc:1428] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 02:59:04.237671 11920 heartbeater.cc:503] Master 127.0.98.62:34995 was elected leader, sending a full tablet report...
I20260924 02:59:04.237833 12073 raft_consensus.cc:348] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3e1a95535049ffabe4b485a96061f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38831 } } peers { permanent_uuid: "8718a7dc82624a97a3f7e4f7b5157e03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43233 } } peers { permanent_uuid: "4f3c46f0b3f341d6aa0350937774a75a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42765 } }
I20260924 02:59:04.237974 12073 raft_consensus.cc:374] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:04.238020 12073 raft_consensus.cc:729] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc3e1a95535049ffabe4b485a96061f6, State: Initialized, Role: FOLLOWER
I20260924 02:59:04.238209 12073 consensus_queue.cc:260] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3e1a95535049ffabe4b485a96061f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38831 } } peers { permanent_uuid: "8718a7dc82624a97a3f7e4f7b5157e03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43233 } } peers { permanent_uuid: "4f3c46f0b3f341d6aa0350937774a75a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42765 } }
I20260924 02:59:04.238466 12073 ts_tablet_manager.cc:1428] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 02:59:04.238489 11781 heartbeater.cc:503] Master 127.0.98.62:34995 was elected leader, sending a full tablet report...
I20260924 02:59:04.319598 12081 raft_consensus.cc:482] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:59:04.319765 12081 raft_consensus.cc:504] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3e1a95535049ffabe4b485a96061f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38831 } } peers { permanent_uuid: "8718a7dc82624a97a3f7e4f7b5157e03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43233 } } peers { permanent_uuid: "4f3c46f0b3f341d6aa0350937774a75a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42765 } }
I20260924 02:59:04.320151 12081 leader_election.cc:288] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8718a7dc82624a97a3f7e4f7b5157e03 (127.0.98.2:43233), 4f3c46f0b3f341d6aa0350937774a75a (127.0.98.3:42765)
I20260924 02:59:04.323827 12080 raft_consensus.cc:482] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:59:04.323944 12080 raft_consensus.cc:504] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3e1a95535049ffabe4b485a96061f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38831 } } peers { permanent_uuid: "8718a7dc82624a97a3f7e4f7b5157e03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43233 } } peers { permanent_uuid: "4f3c46f0b3f341d6aa0350937774a75a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42765 } }
I20260924 02:59:04.324312 12080 leader_election.cc:288] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc3e1a95535049ffabe4b485a96061f6 (127.0.98.1:38831), 4f3c46f0b3f341d6aa0350937774a75a (127.0.98.3:42765)
I20260924 02:59:04.325013 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.320712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54357 (local address 127.0.98.3:42765)
0924 02:59:04.320758 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:04.320760 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:04.321542 (+ 782us) server_negotiation.cc:407] Connection header received
0924 02:59:04.321671 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:04.321673 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:04.321685 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:04.321717 (+ 32us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:04.321940 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.322689 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.324116 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.324639 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.324683 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.324695 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:04.324891 (+ 196us) server_negotiation.cc:299] Negotiation successful
0924 02:59:04.324934 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 02:59:04.325196 12082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.320516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42765 (local address 127.0.98.1:54357)
0924 02:59:04.321470 (+ 954us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:04.321479 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:04.321568 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:04.321783 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:04.321784 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:04.321786 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:04.321865 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:04.321867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.322799 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.322799 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:04.324021 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:04.324024 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.324750 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.324750 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:04.324816 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.324819 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:59:04.325078 (+ 259us) client_negotiation.cc:240] Negotiation successful
0924 02:59:04.325118 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":931,"thread_start_us":91,"threads_started":1}
I20260924 02:59:04.325577 11999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1a13d78456642e986bdcaefdefdc78d" candidate_uuid: "fc3e1a95535049ffabe4b485a96061f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3c46f0b3f341d6aa0350937774a75a" is_pre_election: true
I20260924 02:59:04.325744 11999 raft_consensus.cc:2454] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc3e1a95535049ffabe4b485a96061f6 in term 0.
I20260924 02:59:04.325989 11671 leader_election.cc:302] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f3c46f0b3f341d6aa0350937774a75a, fc3e1a95535049ffabe4b485a96061f6; no voters:
I20260924 02:59:04.326166 12081 raft_consensus.cc:2781] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:59:04.326236 12081 raft_consensus.cc:482] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:59:04.326257 12081 raft_consensus.cc:3037] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:04.327205 12081 raft_consensus.cc:504] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3e1a95535049ffabe4b485a96061f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38831 } } peers { permanent_uuid: "8718a7dc82624a97a3f7e4f7b5157e03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43233 } } peers { permanent_uuid: "4f3c46f0b3f341d6aa0350937774a75a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42765 } }
I20260924 02:59:04.327351 12081 leader_election.cc:288] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [CANDIDATE]: Term 1 election: Requested vote from peers 8718a7dc82624a97a3f7e4f7b5157e03 (127.0.98.2:43233), 4f3c46f0b3f341d6aa0350937774a75a (127.0.98.3:42765)
I20260924 02:59:04.327533 11999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1a13d78456642e986bdcaefdefdc78d" candidate_uuid: "fc3e1a95535049ffabe4b485a96061f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3c46f0b3f341d6aa0350937774a75a"
I20260924 02:59:04.327602 11999 raft_consensus.cc:3037] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:04.328459 11999 raft_consensus.cc:2454] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc3e1a95535049ffabe4b485a96061f6 in term 1.
I20260924 02:59:04.328568 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.324519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38831 (local address 127.0.98.2:42551)
0924 02:59:04.324563 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:04.324575 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:04.324624 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:04.324778 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:04.324779 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:04.324781 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:04.325488 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:04.325490 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.326311 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.326312 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:04.327557 (+ 1245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:04.327560 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.328327 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.328328 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:04.328404 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.328408 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:59:04.328451 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 02:59:04.328497 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23,"mutex_wait_us":652}
I20260924 02:59:04.328629 11671 leader_election.cc:302] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4f3c46f0b3f341d6aa0350937774a75a, fc3e1a95535049ffabe4b485a96061f6; no voters:
I20260924 02:59:04.328702 12081 raft_consensus.cc:2781] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:59:04.328835 12081 raft_consensus.cc:686] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [term 1 LEADER]: Becoming Leader. State: Replica: fc3e1a95535049ffabe4b485a96061f6, State: Running, Role: LEADER
I20260924 02:59:04.328919 12081 consensus_queue.cc:237] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3e1a95535049ffabe4b485a96061f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38831 } } peers { permanent_uuid: "8718a7dc82624a97a3f7e4f7b5157e03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43233 } } peers { permanent_uuid: "4f3c46f0b3f341d6aa0350937774a75a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42765 } }
I20260924 02:59:04.329030 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.324619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42551 (local address 127.0.98.1:38831)
0924 02:59:04.324660 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:04.324662 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:04.324668 (+ 6us) server_negotiation.cc:407] Connection header received
0924 02:59:04.324687 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:04.324687 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:04.324696 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:04.324742 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:04.325565 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.326224 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.327688 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.328226 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.328289 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.328297 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:04.328927 (+ 630us) server_negotiation.cc:299] Negotiation successful
0924 02:59:04.328968 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 02:59:04.329245 11726 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1a13d78456642e986bdcaefdefdc78d" candidate_uuid: "8718a7dc82624a97a3f7e4f7b5157e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc3e1a95535049ffabe4b485a96061f6" is_pre_election: true
I20260924 02:59:04.330595 11570 catalog_manager.cc:5495] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc3e1a95535049ffabe4b485a96061f6 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fc3e1a95535049ffabe4b485a96061f6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc3e1a95535049ffabe4b485a96061f6" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38831 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8718a7dc82624a97a3f7e4f7b5157e03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4f3c46f0b3f341d6aa0350937774a75a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42765 } health_report { overall_health: UNKNOWN } } }
I20260924 02:59:04.332172 12083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.320372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43233 (local address 127.0.98.1:60697)
0924 02:59:04.321254 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:04.321265 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:04.321356 (+ 91us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:04.321750 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:04.321751 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:04.321753 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:04.321844 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:04.321846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.322810 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.322811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:04.324063 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:04.324065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.331884 (+ 7819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.331885 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:04.331971 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.331975 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:59:04.332026 (+ 51us) client_negotiation.cc:240] Negotiation successful
0924 02:59:04.332067 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":860,"mutex_wait_us":22,"thread_start_us":104,"threads_started":1}
I20260924 02:59:04.332789 12072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.320601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60697 (local address 127.0.98.2:43233)
0924 02:59:04.320640 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:04.320643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:04.321583 (+ 940us) server_negotiation.cc:407] Connection header received
0924 02:59:04.321617 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:04.321618 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:04.321630 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:04.321690 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:04.321928 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.322701 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.324705 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.325235 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.325284 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.325293 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:04.332683 (+ 7390us) server_negotiation.cc:299] Negotiation successful
0924 02:59:04.332729 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 02:59:04.333052 11874 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1a13d78456642e986bdcaefdefdc78d" candidate_uuid: "fc3e1a95535049ffabe4b485a96061f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8718a7dc82624a97a3f7e4f7b5157e03" is_pre_election: true
I20260924 02:59:04.333148 11874 raft_consensus.cc:2454] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc3e1a95535049ffabe4b485a96061f6 in term 0.
I20260924 02:59:04.333235 11873 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1a13d78456642e986bdcaefdefdc78d" candidate_uuid: "fc3e1a95535049ffabe4b485a96061f6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8718a7dc82624a97a3f7e4f7b5157e03"
I20260924 02:59:04.333276 11873 raft_consensus.cc:3037] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:04.334067 11873 raft_consensus.cc:2454] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc3e1a95535049ffabe4b485a96061f6 in term 1.
I20260924 02:59:04.334724 11922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.324729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:42765 (local address 127.0.98.2:58897)
0924 02:59:04.328650 (+ 3921us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:04.328661 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:04.328702 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:04.329196 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:04.329197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:04.329199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:04.329231 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:04.329232 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.330743 (+ 1511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.330744 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:04.331743 (+ 999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:04.331746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.334465 (+ 2719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.334466 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:04.334569 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.334573 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:59:04.334623 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 02:59:04.334659 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3897,"thread_start_us":91,"threads_started":1}
I20260924 02:59:04.335099 12070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.325333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:58897 (local address 127.0.98.3:42765)
0924 02:59:04.325375 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:04.325378 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:04.328810 (+ 3432us) server_negotiation.cc:407] Connection header received
0924 02:59:04.328831 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:04.328831 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:04.328839 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:04.328879 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:04.329309 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.330048 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.331839 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.332375 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.332439 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.332448 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:04.334994 (+ 2546us) server_negotiation.cc:299] Negotiation successful
0924 02:59:04.335037 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 02:59:04.335242 11999 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1a13d78456642e986bdcaefdefdc78d" candidate_uuid: "8718a7dc82624a97a3f7e4f7b5157e03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4f3c46f0b3f341d6aa0350937774a75a" is_pre_election: true
I20260924 02:59:04.335321 11999 raft_consensus.cc:2379] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8718a7dc82624a97a3f7e4f7b5157e03 in current term 1: Already voted for candidate fc3e1a95535049ffabe4b485a96061f6 in this term.
I20260924 02:59:04.335537 11810 leader_election.cc:302] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8718a7dc82624a97a3f7e4f7b5157e03; no voters: 4f3c46f0b3f341d6aa0350937774a75a, fc3e1a95535049ffabe4b485a96061f6
I20260924 02:59:04.335702 12080 raft_consensus.cc:2726] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260924 02:59:04.341387 12071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.338133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39830 (local address 127.0.98.1:38831)
0924 02:59:04.338286 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:04.338288 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:04.338293 (+ 5us) server_negotiation.cc:407] Connection header received
0924 02:59:04.338321 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:04.338322 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:04.338338 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:04.338393 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:59:04.338563 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.339281 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.340371 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:04.340665 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:04.340717 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:04.340886 (+ 169us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:59:04.341016 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:59:04.341123 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:04.341281 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 02:59:04.341300 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137}
I20260924 02:59:04.343870 11999 raft_consensus.cc:1261] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a [term 1 FOLLOWER]: Refusing update from remote peer fc3e1a95535049ffabe4b485a96061f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:59:04.344179 12085 consensus_queue.cc:1035] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4f3c46f0b3f341d6aa0350937774a75a" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 42765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:59:04.346355 11873 raft_consensus.cc:1261] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03 [term 1 FOLLOWER]: Refusing update from remote peer fc3e1a95535049ffabe4b485a96061f6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:59:04.346653 12085 consensus_queue.cc:1035] T e1a13d78456642e986bdcaefdefdc78d P fc3e1a95535049ffabe4b485a96061f6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8718a7dc82624a97a3f7e4f7b5157e03" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:59:04.351040 12089 mvcc.cc:205] Tried to move back new op lower bound from 7332735976829947904 to 7332735976771887104. Current Snapshot: MvccSnapshot[applied={T|T < 7332735976829947904}]
I20260924 02:59:04.351246 12091 mvcc.cc:205] Tried to move back new op lower bound from 7332735976829947904 to 7332735976771887104. Current Snapshot: MvccSnapshot[applied={T|T < 7332735976829947904}]
I20260924 02:59:04.353933 11570 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:59:04.354043 11570 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52064:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:59:04.354853 11570 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 53893747257740b490798f7bb9602427: Sending DeleteTablet for 3 replicas of tablet e1a13d78456642e986bdcaefdefdc78d
I20260924 02:59:04.355206 11989 tablet_service.cc:1514] Processing DeleteTablet for tablet e1a13d78456642e986bdcaefdefdc78d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:59:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:46858
I20260924 02:59:04.355645 11715 tablet_service.cc:1514] Processing DeleteTablet for tablet e1a13d78456642e986bdcaefdefdc78d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:59:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:39822
I20260924 02:59:04.355842 11854 tablet_service.cc:1514] Processing DeleteTablet for tablet e1a13d78456642e986bdcaefdefdc78d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:59:04 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51000
I20260924 02:59:04.357259 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 11652
I20260924 02:59:04.358808 12099 tablet_replica.cc:330] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03: stopping tablet replica
I20260924 02:59:04.358911 12099 raft_consensus.cc:2229] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:59:04.359020 12099 raft_consensus.cc:2258] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:59:04.359171 12097 tablet_replica.cc:330] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a: stopping tablet replica
I20260924 02:59:04.359259 12097 raft_consensus.cc:2229] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:59:04.359349 12097 raft_consensus.cc:2258] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:59:04.360117 12097 ts_tablet_manager.cc:1905] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:59:04.361836 12097 ts_tablet_manager.cc:1918] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:59:04.361902 12097 log.cc:1196] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/e1a13d78456642e986bdcaefdefdc78d
I20260924 02:59:04.362494 12097 ts_tablet_manager.cc:1939] T e1a13d78456642e986bdcaefdefdc78d P 4f3c46f0b3f341d6aa0350937774a75a: Deleting consensus metadata
I20260924 02:59:04.362926 11558 catalog_manager.cc:4841] TS 4f3c46f0b3f341d6aa0350937774a75a (127.0.98.3:42765): tablet e1a13d78456642e986bdcaefdefdc78d (table test-table [id=1f992eb165e642c4849f799acdaa50b0]) successfully deleted
I20260924 02:59:04.363294 12099 ts_tablet_manager.cc:1905] T e1a13d78456642e986bdcaefdefdc78d P 8718a7dc82624a97a3f7e4f7b5157e03: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260924 02:59:04.364008 11556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38831 (error 108)
I20260924 02:59:04.364840 12069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:04.364707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38831 (local address 127.0.0.1:39840)
0924 02:59:04.364756 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:04.364805 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 02:59:04.364850 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 11789
W20260924 02:59:04.366284 11556 catalog_manager.cc:4584] TS fc3e1a95535049ffabe4b485a96061f6 (127.0.98.1:38831): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e1a13d78456642e986bdcaefdefdc78d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38831: connect: Connection refused (error 111)
W20260924 02:59:04.370847 12069 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:59:04.370610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43233 (local address 127.0.0.1:51010)
0924 02:59:04.370666 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:04.370680 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:04.370746 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:04.370797 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:43233: BlockingWrite error: write error: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 02:59:04.370993 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 11925
I20260924 02:59:04.376261 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 11539
I20260924 02:59:04.382205 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 02:59:04.716325 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 02:59:04.716420 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 02:59:05.050299 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 02:59:05.050 UTC [11125] LOG: received smart shutdown request
2026-09-24 02:59:05.053 UTC [11125] LOG: background worker "logical replication launcher" (PID 11134) exited with exit code 1
2026-09-24 02:59:05.053 UTC [11129] LOG: shutting down
2026-09-24 02:59:05.206 UTC [11125] LOG: database system is shut down
2026-09-24T02:59:05Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43518 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:59:05 dist-test-slave-sb6p krb5kdc[12109](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:59:05 dist-test-slave-sb6p krb5kdc[12109](info): set up 2 sockets
Sep 24 02:59:05 dist-test-slave-sb6p krb5kdc[12109](info): commencing operation
krb5kdc: starting...
W20260924 02:59:06.238401 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:59:06 dist-test-slave-sb6p krb5kdc[12109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218746, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:59:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:59:06Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/rangeradmin_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/rangerlookup_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
I20260924 02:59:06.385349 392 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskPoI4uN/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/postgres -l logfile start
2026-09-24 02:59:08.675 UTC [12139] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-24 02:59:08.675 UTC [12139] LOG: listening on IPv4 address "127.0.98.20", port 44381
2026-09-24 02:59:08.679 UTC [12139] LOG: listening on Unix socket "/tmp/.s.PGSQL.44381"
2026-09-24 02:59:08.706 UTC [12142] LOG: database system was shut down at 2026-09-24 02:59:07 UTC
2026-09-24 02:59:08.709 UTC [12139] LOG: database system is ready to accept connections
I20260924 02:59:09.570122 392 mini_postgres.cc:96] Postgres bound to 44381
2026-09-24 02:59:09.575 UTC [12150] FATAL: database "slave" does not exist
127.0.98.20:44381 - accepting connections
I20260924 02:59:09.576035 392 mini_ranger.cc:161] Starting Ranger...
I20260924 02:59:09.589572 392 mini_ranger.cc:84] Created miniranger Postgres user
I20260924 02:59:10.034608 392 mini_ranger.cc:87] Created ranger Postgres database
I20260924 02:59:10.034674 392 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-admin
2026-09-24 02:59:10,276 [I] DB FLAVOR :POSTGRES
2026-09-24 02:59:10,276 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 02:59:10,277 [I] Checking connection..
2026-09-24 02:59:10,277 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-24 02:59:10,512 [I] Checking connection passed.
2026-09-24 02:59:10,513 [I] --------- Verifying version history table ---------
2026-09-24 02:59:10,513 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 02:59:10,740 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 02:59:10,740 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 02:59:10,962 [I] Table x_db_version_h does not exist in database ranger
2026-09-24 02:59:10,963 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-24 02:59:10,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-24 02:59:11.210 UTC [12214] WARNING: there is no transaction in progress
2026-09-24 02:59:11,216 [I] create_dbversion_catalog.sql file imported successfully
2026-09-24 02:59:11,216 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-24 02:59:11,455 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-24 02:59:11,455 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 02:59:11,455 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-24 02:59:11,685 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-24 02:59:11,924 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-sb6p.c.gcp-upstream.internal','N') ;"
2026-09-24 02:59:12,149 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 02:59:12,393 [I] Table x_portal_user does not exist in database ranger
2026-09-24 02:59:12,394 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 02:59:12,636 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-24 02:59:12,636 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 02:59:12,888 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 02:59:13,111 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-24 02:59:13,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-24 02:59:14.022 UTC [12322] WARNING: there is no transaction in progress
2026-09-24 02:59:14.038 UTC [12322] WARNING: there is no transaction in progress
2026-09-24 02:59:14.054 UTC [12322] WARNING: there is no transaction in progress
2026-09-24 02:59:14.070 UTC [12322] WARNING: there is no transaction in progress
2026-09-24 02:59:14.089 UTC [12322] WARNING: there is no transaction in progress
2026-09-24 02:59:14.109 UTC [12322] WARNING: there is no transaction in progress
2026-09-24 02:59:14.175 UTC [12322] WARNING: there is no transaction in progress
2026-09-24 02:59:14.181 UTC [12322] WARNING: there is no transaction in progress
2026-09-24 02:59:14.194 UTC [12322] WARNING: there is no transaction in progress
2026-09-24 02:59:14.204 UTC [12322] WARNING: there is no transaction in progress
2026-09-24 02:59:14.213 UTC [12322] WARNING: there is no transaction in progress
2026-09-24 02:59:14.227 UTC [12322] WARNING: there is no transaction in progress
2026-09-24 02:59:14.236 UTC [12322] WARNING: there is no transaction in progress
2026-09-24 02:59:14.242 UTC [12322] WARNING: there is no transaction in progress
2026-09-24 02:59:14.248 UTC [12322] WARNING: there is no transaction in progress
2026-09-24 02:59:14.254 UTC [12322] WARNING: there is no transaction in progress
2026-09-24 02:59:14,864 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-24 02:59:14,864 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-24 02:59:15,084 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-24 02:59:15,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-24 02:59:15,332 [I] Table x_portal_user already exists in database 'ranger'
2026-09-24 02:59:15,333 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-24 02:59:15,587 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-24 02:59:15,587 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-24 02:59:15,837 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-24 02:59:16,086 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-sb6p.c.gcp-upstream.internal';"
2026-09-24 02:59:16,322 [I] CORE_DB_SCHEMA import status has been updated
2026-09-24 02:59:16,322 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-24 02:59:16,574 [I] DB_PATCHES have already been applied
I20260924 02:59:16.579331 392 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-admin:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskPoI4uN/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260924 02:59:16.579393 392 mini_ranger.cc:193] Using host: 127.0.98.20
I20260924 02:59:16.582047 392 mini_ranger.cc:227] Ranger admin URL: http://127.0.98.20:37953
Sep 24, 2026 2:59:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 2:59:16 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 2:59:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-admin/ews/webapp
Sep 24, 2026 2:59:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 24, 2026 2:59:16 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 24, 2026 2:59:17 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-admin/ews/webapp].
Sep 24, 2026 2:59:17 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260924 02:59:17.573176 392 mini_ranger.cc:160] Time spent starting Ranger: real 7.997s user 0.000s sys 0.007s
Sep 24, 2026 2:59:36 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 24, 2026 2:59:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 24, 2026 2:59:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 24, 2026 2:59:37 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 24, 2026 2:59:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 24, 2026 2:59:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 24, 2026 2:59:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 24, 2026 2:59:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 24, 2026 2:59:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 24, 2026 2:59:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 24, 2026 2:59:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 24, 2026 2:59:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 24, 2026 2:59:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 24, 2026 2:59:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 24, 2026 2:59:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 24, 2026 2:59:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 24, 2026 2:59:37 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 24, 2026 2:59:37 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260924 02:59:38.498742 392 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.98.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.98.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.98.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/rangerkms_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.98.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.98.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.98.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/HTTP_127.0.98.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 24 02:59:38 dist-test-slave-sb6p krb5kdc[12109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218778, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260924 02:59:38.574949 392 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260924 02:59:38.583011 392 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260924 02:59:39.413100 392 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260924 02:59:39.413179 392 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-kms
I20260924 02:59:39.413194 392 mini_ranger_kms.cc:227] Using postgres at 127.0.98.20:44381
2026-09-24 02:59:39,558 [I] DB FLAVOR :POSTGRES
2026-09-24 02:59:39,558 [I] --------- Verifying Ranger DB connection ---------
2026-09-24 02:59:39,559 [I] Checking connection
2026-09-24 02:59:39,559 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-24 02:59:39,777 [I] connection success
2026-09-24 02:59:39,778 [I] --------- Verifying Ranger DB tables ---------
2026-09-24 02:59:39,778 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-24 02:59:39,778 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-24 02:59:40,000 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-24 02:59:40,001 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-24 02:59:40,001 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-24 02:59:40,001 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.98.20:44381/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-24 02:59:40,271 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260924 02:59:40.754035 392 mini_ranger_kms.cc:310] Created kms service
I20260924 02:59:40.956779 392 mini_ranger_kms.cc:326] Created kudu user
I20260924 02:59:40.997079 392 mini_ranger_kms.cc:343] Created rangerkms user
Sep 24 02:59:40 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218778, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 02:59:41.402362 392 mini_ranger_kms.cc:384] Added ranger policy
I20260924 02:59:41.402472 392 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-kms:/tmp/dist-test-taskPoI4uN/build/release/bin/postgresql.jar:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskPoI4uN/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260924 02:59:41.402491 392 mini_ranger_kms.cc:240] Using host: 127.0.98.20
I20260924 02:59:41.405673 392 mini_ranger_kms.cc:276] Ranger KMS PID: 12502
I20260924 02:59:41.405758 392 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.98.20:40309
Sep 24, 2026 2:59:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 24, 2026 2:59:42 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 24, 2026 2:59:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 24, 2026 2:59:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 24, 2026 2:59:42 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-kms/ews/webapp].
Sep 24, 2026 2:59:42 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260924 02:59:43.403703 392 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.829s user 0.000s sys 0.129s
I20260924 02:59:43.403785 392 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260924 02:59:43.403826 392 mini_ranger_kms.cc:401] 127.0.98.20:40309/kms/v1/keys
Sep 24 02:59:43 dist-test-slave-sb6p krb5kdc[12109](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790218783, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 02:59:43 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218783, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 02:59:44 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218783, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.98.20@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
Sep 24 02:59:46 dist-test-slave-sb6p krb5kdc[12109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 02:59:46.434372 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40093
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40093
--ranger_config_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40309/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:46.534030 12553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:46.534231 12553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:46.534250 12553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:46.535363 12553 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:59:46.535399 12553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:46.535413 12553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:46.535426 12553 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:59:46.535441 12553 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:59:46.536991 12553 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.98.20:40309/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40093
--ranger_config_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:46.537170 12553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:46.537338 12553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:46.542636 12561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:46.542681 12558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:46.542646 12559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:46.542644 12553 server_base.cc:1034] running on GCE node
I20260924 02:59:46.543125 12553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:46.543377 12553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:46.544503 12553 hybrid_clock.cc:648] HybridClock initialized: now 1790218786544491 us; error 27 us; skew 500 ppm
Sep 24 02:59:46 dist-test-slave-sb6p krb5kdc[12109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:46.546881 12553 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:59:46.547348 12553 webserver.cc:460] Webserver started at http://127.0.98.62:42681/ using document root <none> and password file <none>
I20260924 02:59:46.547549 12553 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:46.547591 12553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:46.547708 12553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 02:59:46 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 02:59:46.599618 12553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "79d6a3f118b74dafb186cfe9b9e99bf1"
format_stamp: "Formatted at 2026-09-24 02:59:46 on dist-test-slave-sb6p"
server_key: "88bba376e28b1c1042b853eb526e8088"
server_key_iv: "0aaaa741360da3e7351cbfbcd3103fe9"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:46.599982 12553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "79d6a3f118b74dafb186cfe9b9e99bf1"
format_stamp: "Formatted at 2026-09-24 02:59:46 on dist-test-slave-sb6p"
server_key: "88bba376e28b1c1042b853eb526e8088"
server_key_iv: "0aaaa741360da3e7351cbfbcd3103fe9"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:46.601255 12553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:59:46.615785 12568 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:46.615959 12553 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:59:46.616051 12553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "79d6a3f118b74dafb186cfe9b9e99bf1"
format_stamp: "Formatted at 2026-09-24 02:59:46 on dist-test-slave-sb6p"
server_key: "88bba376e28b1c1042b853eb526e8088"
server_key_iv: "0aaaa741360da3e7351cbfbcd3103fe9"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:46.616158 12553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:46.630127 12553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:46.631943 12553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:46.632091 12553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:46.635849 12553 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40093
I20260924 02:59:46.635891 12620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40093 every 8 connection(s)
I20260924 02:59:46.636217 12553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:59:46.636835 12621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:46.639046 12621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1: Bootstrap starting.
I20260924 02:59:46.639583 12621 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:46.639832 12621 log.cc:824] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:46.640410 12621 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1: No bootstrap required, opened a new log
I20260924 02:59:46.640462 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 12553
I20260924 02:59:46.640559 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:59:46.640738 392 external_mini_cluster.cc:1428] Setting key a291895cc8a1363a689279c17844aaa2
I20260924 02:59:46.642719 12621 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d6a3f118b74dafb186cfe9b9e99bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40093 } }
I20260924 02:59:46.642856 12621 raft_consensus.cc:374] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:46.642889 12621 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d6a3f118b74dafb186cfe9b9e99bf1, State: Initialized, Role: FOLLOWER
I20260924 02:59:46.643044 12621 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d6a3f118b74dafb186cfe9b9e99bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40093 } }
I20260924 02:59:46.643160 12621 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:59:46.643216 12621 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:59:46.643297 12621 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:46.643927 12621 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d6a3f118b74dafb186cfe9b9e99bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40093 } }
I20260924 02:59:46.644021 12621 leader_election.cc:302] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79d6a3f118b74dafb186cfe9b9e99bf1; no voters:
I20260924 02:59:46.644188 12621 leader_election.cc:288] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:59:46.644230 12626 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:59:46.644405 12626 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [term 1 LEADER]: Becoming Leader. State: Replica: 79d6a3f118b74dafb186cfe9b9e99bf1, State: Running, Role: LEADER
I20260924 02:59:46.644487 12621 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:59:46.644523 12626 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d6a3f118b74dafb186cfe9b9e99bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40093 } }
I20260924 02:59:46.644936 12626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "79d6a3f118b74dafb186cfe9b9e99bf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d6a3f118b74dafb186cfe9b9e99bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40093 } } }
I20260924 02:59:46.645092 12626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [sys.catalog]: This master's current role is: LEADER
I20260924 02:59:46.645251 12626 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79d6a3f118b74dafb186cfe9b9e99bf1. Latest consensus state: current_term: 1 leader_uuid: "79d6a3f118b74dafb186cfe9b9e99bf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d6a3f118b74dafb186cfe9b9e99bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40093 } } }
I20260924 02:59:46.645880 12621 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 02:59:46.646354 12626 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [sys.catalog]: This master's current role is: LEADER
I20260924 02:59:46.648314 12635 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:59:46.649976 12635 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 24 02:59:46 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218786, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:46.655208 12625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:46.641266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34172 (local address 127.0.98.62:40093)
0924 02:59:46.642645 (+ 1379us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:46.642655 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:46.642690 (+ 35us) server_negotiation.cc:407] Connection header received
0924 02:59:46.642820 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:46.642828 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:46.642883 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:46.643025 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:46.643777 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:46.645257 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:46.646324 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:46.648705 (+ 2381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:46.648872 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:46.651617 (+ 2745us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:46.651620 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:46.651621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:46.651632 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:46.653468 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:46.653858 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:46.653861 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:46.653861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:46.653966 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:46.654248 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:46.654249 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:46.654249 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:46.654592 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:46.654741 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:46.654854 (+ 113us) server_negotiation.cc:299] Negotiation successful
0924 02:59:46.654984 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1327,"thread_start_us":69,"threads_started":1}
I20260924 02:59:46.658807 12635 catalog_manager.cc:1347] Generated new cluster ID: d33a67c71a624f6182f4df8775437d03
I20260924 02:59:46.658860 12635 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:59:46.666643 12635 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:59:46.667145 12635 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:59:46.673359 12635 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1: Generated new TSK 0
I20260924 02:59:46.673528 12635 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 02:59:47.371119 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40093
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:47.485847 12666 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:47.486061 12666 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:47.486089 12666 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:47.487277 12666 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:47.487321 12666 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:47.487346 12666 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:59:47.488870 12666 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.98.20:40309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12666
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:47.489113 12666 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:47.489342 12666 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:47.494086 12672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:47.494278 12674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:47.494372 12671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:47.494531 12666 server_base.cc:1034] running on GCE node
I20260924 02:59:47.494719 12666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:47.494963 12666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:47.498207 12666 hybrid_clock.cc:648] HybridClock initialized: now 1790218787498183 us; error 32 us; skew 500 ppm
Sep 24 02:59:47 dist-test-slave-sb6p krb5kdc[12109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:47.500671 12666 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 02:59:47.501050 12666 webserver.cc:460] Webserver started at http://127.0.98.1:36435/ using document root <none> and password file <none>
I20260924 02:59:47.501250 12666 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:47.501297 12666 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:47.501420 12666 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 02:59:47 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 02:59:47.514050 12666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "19a40e9cd73f476db31d6bb26586bdbf"
format_stamp: "Formatted at 2026-09-24 02:59:47 on dist-test-slave-sb6p"
server_key: "715f5474189abc6c3944aca28f4e50f8"
server_key_iv: "c403c32080a794eb59351618f9b009fb"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:47.514482 12666 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "19a40e9cd73f476db31d6bb26586bdbf"
format_stamp: "Formatted at 2026-09-24 02:59:47 on dist-test-slave-sb6p"
server_key: "715f5474189abc6c3944aca28f4e50f8"
server_key_iv: "c403c32080a794eb59351618f9b009fb"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:47.517887 12666 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260924 02:59:47.535017 12683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:47.535260 12666 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 02:59:47.535346 12666 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "19a40e9cd73f476db31d6bb26586bdbf"
format_stamp: "Formatted at 2026-09-24 02:59:47 on dist-test-slave-sb6p"
server_key: "715f5474189abc6c3944aca28f4e50f8"
server_key_iv: "c403c32080a794eb59351618f9b009fb"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:47.535437 12666 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:47.562642 12666 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:47.564425 12666 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:47.564560 12666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:47.564740 12666 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:47.565060 12666 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:59:47.565102 12666 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:47.565131 12666 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:59:47.565172 12666 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:47.571408 12666 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34079
I20260924 02:59:47.571758 12666 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:59:47.572121 12796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34079 every 8 connection(s)
I20260924 02:59:47.575423 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 12666
I20260924 02:59:47.575513 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:59:47.575673 392 external_mini_cluster.cc:1428] Setting key 5b757e5e32b09646136e8688a5647ad2
Sep 24 02:59:47 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:47.580744 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:47.572508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:35513 (local address 127.0.98.62:40093)
0924 02:59:47.573350 (+ 842us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:47.573351 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:47.573358 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:59:47.573379 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:47.573380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:47.573389 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:47.573442 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:47.573982 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:47.574479 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:47.575580 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:47.575855 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:47.575909 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:47.578153 (+ 2244us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:47.578155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:47.578155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:47.578162 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:47.579510 (+ 1348us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:47.579857 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:47.579858 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:47.579858 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:47.579915 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:47.580123 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:47.580124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:47.580124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:47.580226 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:47.580280 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:47.580537 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 02:59:47.580617 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":74,"threads_started":1}
I20260924 02:59:47.580866 12800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:47.572502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40093 (local address 127.0.98.1:35513)
0924 02:59:47.572647 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:47.572687 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:47.573214 (+ 527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:47.573534 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:47.573539 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:47.573714 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:47.573896 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:47.573900 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:47.574592 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:47.574593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:47.575474 (+ 881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:47.575478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:47.575985 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:47.575986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:47.576118 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:47.576635 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:47.576645 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:47.578068 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:47.579588 (+ 1520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:47.579594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:47.579596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:47.579795 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:47.579966 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:47.579967 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:47.579967 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:47.580070 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:47.580277 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:47.580283 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:47.580424 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 02:59:47.580510 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 02:59:47.580655 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":74,"threads_started":1}
I20260924 02:59:47.581370 12797 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40093
I20260924 02:59:47.581459 12797 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:47.581624 12797 heartbeater.cc:511] Master 127.0.98.62:40093 requested a full tablet report, sending...
I20260924 02:59:47.582096 12585 ts_manager.cc:194] Registered new tserver with Master: 19a40e9cd73f476db31d6bb26586bdbf (127.0.98.1:34079)
I20260924 02:59:47.582734 12585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:35513
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 02:59:47.604915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40093
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:47.699025 12803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:47.699301 12803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:47.699342 12803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:47.700825 12803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:47.700924 12803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:47.700989 12803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:59:47.702827 12803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.98.20:40309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12803
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:47.703083 12803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:47.703311 12803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:47.708369 12811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:47.708567 12803 server_base.cc:1034] running on GCE node
W20260924 02:59:47.708695 12810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:47.708773 12813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:47.709020 12803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:47.709245 12803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:47.710373 12803 hybrid_clock.cc:648] HybridClock initialized: now 1790218787710360 us; error 23 us; skew 500 ppm
Sep 24 02:59:47 dist-test-slave-sb6p krb5kdc[12109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:47.712771 12803 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 02:59:47.713202 12803 webserver.cc:460] Webserver started at http://127.0.98.2:40417/ using document root <none> and password file <none>
I20260924 02:59:47.713415 12803 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:47.713519 12803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:47.713698 12803 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 02:59:47 dist-test-slave-sb6p krb5kdc[12109](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790218787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 24 02:59:47 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 02:59:47.731317 12803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "fec5cc6aa5dd4b62af664dc58728d9d1"
format_stamp: "Formatted at 2026-09-24 02:59:47 on dist-test-slave-sb6p"
server_key: "29cb33b098e332efb28a8a7520c0fa2d"
server_key_iv: "1625953806029e2ac9bad02a29c7a051"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:47.732357 12803 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "fec5cc6aa5dd4b62af664dc58728d9d1"
format_stamp: "Formatted at 2026-09-24 02:59:47 on dist-test-slave-sb6p"
server_key: "29cb33b098e332efb28a8a7520c0fa2d"
server_key_iv: "1625953806029e2ac9bad02a29c7a051"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:47.736313 12803 fs_manager.cc:691] Time spent creating directory manager: real 0.004s user 0.001s sys 0.000s
I20260924 02:59:47.761557 12820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:47.761793 12803 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:59:47.761883 12803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "fec5cc6aa5dd4b62af664dc58728d9d1"
format_stamp: "Formatted at 2026-09-24 02:59:47 on dist-test-slave-sb6p"
server_key: "29cb33b098e332efb28a8a7520c0fa2d"
server_key_iv: "1625953806029e2ac9bad02a29c7a051"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:47.761973 12803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:47.772423 12803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:47.774506 12803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:47.774724 12803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:47.774983 12803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:47.775372 12803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:59:47.775461 12803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:47.775518 12803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:59:47.775560 12803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:47.781913 12803 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46813
I20260924 02:59:47.782210 12933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46813 every 8 connection(s)
I20260924 02:59:47.782385 12803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 02:59:47 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:47.790887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 12803
I20260924 02:59:47.790973 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:59:47.791163 392 external_mini_cluster.cc:1428] Setting key 03e1199ab2c918c598a0a05f0aead007
I20260924 02:59:47.793980 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:47.784159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:35633 (local address 127.0.98.62:40093)
0924 02:59:47.784210 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:47.784213 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:47.784225 (+ 12us) server_negotiation.cc:407] Connection header received
0924 02:59:47.784890 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:47.784893 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:47.784921 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:47.784950 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:47.785752 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:47.786604 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:47.788113 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:47.788426 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:47.788465 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:47.790974 (+ 2509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:47.790975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:47.790976 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:47.790983 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:47.792529 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:47.792984 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:47.792985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:47.792985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:47.793041 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:47.793286 (+ 245us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:47.793287 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:47.793287 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:47.793407 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:47.793444 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:47.793802 (+ 358us) server_negotiation.cc:299] Negotiation successful
0924 02:59:47.793886 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 02:59:47.796851 12936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:47.783877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40093 (local address 127.0.98.2:35633)
0924 02:59:47.784066 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:47.784117 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:47.784724 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:47.785140 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:47.785144 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:47.785377 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:47.785644 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:47.785650 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:47.786709 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:47.786710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:47.788001 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:47.788004 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:47.788562 (+ 558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:47.788562 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:47.788697 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:47.789258 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:47.789269 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:47.790861 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:47.792631 (+ 1770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:47.792638 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:47.792640 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:47.792906 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:47.793106 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:47.793107 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:47.793107 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:47.793224 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:47.793483 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:47.793487 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:47.793662 (+ 175us) client_negotiation.cc:769] Sending connection context
0924 02:59:47.794060 (+ 398us) client_negotiation.cc:240] Negotiation successful
0924 02:59:47.794208 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":92,"threads_started":1}
I20260924 02:59:47.797744 12934 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40093
I20260924 02:59:47.797835 12934 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:47.798012 12934 heartbeater.cc:511] Master 127.0.98.62:40093 requested a full tablet report, sending...
I20260924 02:59:47.798380 12585 ts_manager.cc:194] Registered new tserver with Master: fec5cc6aa5dd4b62af664dc58728d9d1 (127.0.98.2:46813)
I20260924 02:59:47.798880 12585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:35633
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 02:59:47.829262 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40093
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:47.933761 12940 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:47.934082 12940 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:47.934163 12940 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:47.935866 12940 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:47.935922 12940 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:47.935959 12940 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:59:47.937994 12940 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.98.20:40309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12940
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:47.938414 12940 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:47.938715 12940 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:47.944854 12946 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:47.944865 12948 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:47.944851 12945 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:47.945330 12940 server_base.cc:1034] running on GCE node
I20260924 02:59:47.945514 12940 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:47.945770 12940 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:47.948982 12940 hybrid_clock.cc:648] HybridClock initialized: now 1790218787948964 us; error 35 us; skew 500 ppm
Sep 24 02:59:47 dist-test-slave-sb6p krb5kdc[12109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:47.951433 12940 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 02:59:47.951869 12940 webserver.cc:460] Webserver started at http://127.0.98.3:44999/ using document root <none> and password file <none>
I20260924 02:59:47.952067 12940 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:47.952116 12940 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:47.952232 12940 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 24 02:59:47 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 02:59:47.964537 12940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "69fec6b0afa04ace80402ec000dca14d"
format_stamp: "Formatted at 2026-09-24 02:59:47 on dist-test-slave-sb6p"
server_key: "f02be621ca901cf50f9820b454eb2c06"
server_key_iv: "b439d7e67d086128563e4f83ba46ac1e"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:47.965289 12940 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "69fec6b0afa04ace80402ec000dca14d"
format_stamp: "Formatted at 2026-09-24 02:59:47 on dist-test-slave-sb6p"
server_key: "f02be621ca901cf50f9820b454eb2c06"
server_key_iv: "b439d7e67d086128563e4f83ba46ac1e"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:47.967300 12940 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260924 02:59:47.988631 12955 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:47.988924 12940 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.001s
I20260924 02:59:47.989082 12940 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "69fec6b0afa04ace80402ec000dca14d"
format_stamp: "Formatted at 2026-09-24 02:59:47 on dist-test-slave-sb6p"
server_key: "f02be621ca901cf50f9820b454eb2c06"
server_key_iv: "b439d7e67d086128563e4f83ba46ac1e"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:47.989244 12940 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:48.014587 12940 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:48.016849 12940 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:48.016984 12940 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:48.017199 12940 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:48.017513 12940 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:59:48.017546 12940 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:48.017583 12940 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:59:48.017629 12940 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:48.022565 12940 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32807
I20260924 02:59:48.022897 12940 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 02:59:48.026710 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 12940
I20260924 02:59:48.026796 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:59:48.026952 392 external_mini_cluster.cc:1428] Setting key da01cc0be0ba36df25b20a9e7ec1062c
I20260924 02:59:48.034854 13068 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32807 every 8 connection(s)
Sep 24 02:59:48 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:48.046801 12799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:48.035196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52973 (local address 127.0.98.62:40093)
0924 02:59:48.035240 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:48.035242 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:48.035485 (+ 243us) server_negotiation.cc:407] Connection header received
0924 02:59:48.036119 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:48.036120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:48.036133 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:48.036202 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:48.036931 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.037683 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.039305 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.039621 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.039698 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:48.042667 (+ 2969us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:48.042668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:48.042668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:48.042676 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:48.044395 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:48.045698 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:48.045699 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:48.045699 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:48.045759 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:48.046027 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:48.046028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:48.046029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:48.046167 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:48.046208 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:48.046614 (+ 406us) server_negotiation.cc:299] Negotiation successful
0924 02:59:48.046703 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 02:59:48.046952 13071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:48.035087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40093 (local address 127.0.98.3:52973)
0924 02:59:48.035347 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:48.035390 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:48.036005 (+ 615us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:48.036292 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:48.036296 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:48.036557 (+ 261us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:48.036831 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:48.036836 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.037791 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.037792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:48.039213 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:48.039215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.040213 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.040214 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:48.040368 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:48.040956 (+ 588us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:48.040966 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:48.042562 (+ 1596us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:48.045304 (+ 2742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:48.045311 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:48.045313 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:48.045605 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:48.045830 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:48.045831 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:48.045831 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:48.045958 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:48.046252 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:48.046258 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:48.046431 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 02:59:48.046570 (+ 139us) client_negotiation.cc:240] Negotiation successful
0924 02:59:48.046726 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":92,"threads_started":1}
I20260924 02:59:48.047636 13069 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40093
I20260924 02:59:48.047758 13069 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:48.047961 13069 heartbeater.cc:511] Master 127.0.98.62:40093 requested a full tablet report, sending...
I20260924 02:59:48.048388 12585 ts_manager.cc:194] Registered new tserver with Master: 69fec6b0afa04ace80402ec000dca14d (127.0.98.3:32807)
I20260924 02:59:48.048909 12585 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52973
I20260924 02:59:48.049835 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 02:59:48.050589 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 12666
I20260924 02:59:48.056358 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 12803
I20260924 02:59:48.060676 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 12940
I20260924 02:59:48.064913 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 12553
I20260924 02:59:48.069380 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40093
--webserver_interface=127.0.98.62
--webserver_port=42681
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40093
--ranger_config_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40309/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:40309/kms with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:48.146454 13072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:48.146667 13072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:48.146687 13072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:48.147753 13072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:59:48.147785 13072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:48.147797 13072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:48.147809 13072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:59:48.147822 13072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:59:48.149276 13072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:40309/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40093
--ranger_config_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40093
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=42681
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13072
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:48.149443 13072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:48.149614 13072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:48.153910 13080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:48.153978 13078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:48.154076 13072 server_base.cc:1034] running on GCE node
W20260924 02:59:48.154163 13077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:48.154362 13072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:48.154661 13072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:48.155795 13072 hybrid_clock.cc:648] HybridClock initialized: now 1790218788155782 us; error 27 us; skew 500 ppm
Sep 24 02:59:48 dist-test-slave-sb6p krb5kdc[12109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:48.157996 13072 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:59:48.158389 13072 webserver.cc:460] Webserver started at http://127.0.98.62:42681/ using document root <none> and password file <none>
I20260924 02:59:48.158624 13072 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:48.158674 13072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 02:59:48 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 02:59:48.216019 13072 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 02:59:48.216807 13088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:48.216974 13072 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:59:48.217064 13072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "79d6a3f118b74dafb186cfe9b9e99bf1"
format_stamp: "Formatted at 2026-09-24 02:59:46 on dist-test-slave-sb6p"
server_key: "88bba376e28b1c1042b853eb526e8088"
server_key_iv: "0aaaa741360da3e7351cbfbcd3103fe9"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:48.217516 13072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:48.262953 13072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:48.264842 13072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:48.264962 13072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:48.268713 13072 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40093
I20260924 02:59:48.268764 13140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40093 every 8 connection(s)
I20260924 02:59:48.269110 13072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:59:48.270272 13141 sys_catalog.cc:263] Verifying existing consensus state
I20260924 02:59:48.270957 13141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1: Bootstrap starting.
I20260924 02:59:48.274509 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 13072
I20260924 02:59:48.274832 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:34079
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:40093
--webserver_port=36435
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:40309/kms with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 02:59:48.279035 13141 log.cc:824] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:48.280651 13141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 02:59:48.280905 13141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1: Bootstrap complete.
I20260924 02:59:48.283147 13141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d6a3f118b74dafb186cfe9b9e99bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40093 } }
I20260924 02:59:48.283370 13141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 79d6a3f118b74dafb186cfe9b9e99bf1, State: Initialized, Role: FOLLOWER
I20260924 02:59:48.283519 13141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d6a3f118b74dafb186cfe9b9e99bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40093 } }
I20260924 02:59:48.283586 13141 raft_consensus.cc:388] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:59:48.283617 13141 raft_consensus.cc:482] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:59:48.283658 13141 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [term 1 FOLLOWER]: Advancing to term 2
I20260924 02:59:48.284623 13141 raft_consensus.cc:504] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d6a3f118b74dafb186cfe9b9e99bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40093 } }
I20260924 02:59:48.284711 13141 leader_election.cc:302] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 79d6a3f118b74dafb186cfe9b9e99bf1; no voters:
I20260924 02:59:48.284874 13141 leader_election.cc:288] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 02:59:48.284922 13145 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 02:59:48.285075 13145 raft_consensus.cc:686] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [term 2 LEADER]: Becoming Leader. State: Replica: 79d6a3f118b74dafb186cfe9b9e99bf1, State: Running, Role: LEADER
I20260924 02:59:48.285121 13141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:59:48.285449 13145 consensus_queue.cc:237] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d6a3f118b74dafb186cfe9b9e99bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40093 } }
I20260924 02:59:48.285729 13147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "79d6a3f118b74dafb186cfe9b9e99bf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d6a3f118b74dafb186cfe9b9e99bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40093 } } }
I20260924 02:59:48.285822 13147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [sys.catalog]: This master's current role is: LEADER
I20260924 02:59:48.285734 13146 sys_catalog.cc:455] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 79d6a3f118b74dafb186cfe9b9e99bf1. Latest consensus state: current_term: 2 leader_uuid: "79d6a3f118b74dafb186cfe9b9e99bf1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "79d6a3f118b74dafb186cfe9b9e99bf1" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40093 } } }
I20260924 02:59:48.285912 13146 sys_catalog.cc:458] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1 [sys.catalog]: This master's current role is: LEADER
I20260924 02:59:48.286316 13154 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:59:48.286671 13141 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260924 02:59:48.286798 13154 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:59:48.286922 13154 catalog_manager.cc:1259] Loaded cluster ID: d33a67c71a624f6182f4df8775437d03
I20260924 02:59:48.286958 13154 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:59:48.290313 13154 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:59:48.290768 13154 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1: Loaded TSK: 0
I20260924 02:59:48.291018 13154 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 02:59:48.383319 13143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:48.383572 13143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:48.383606 13143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:48.385177 13143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:48.385267 13143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:48.385325 13143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:59:48.387490 13143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:40309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:34079
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=36435
--tserver_master_addrs=127.0.98.62:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13143
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:48.387852 13143 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:48.388166 13143 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:48.394380 13178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:48.394408 13181 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:48.394438 13179 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:48.394600 13143 server_base.cc:1034] running on GCE node
I20260924 02:59:48.394973 13143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:48.395236 13143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:48.396399 13143 hybrid_clock.cc:648] HybridClock initialized: now 1790218788396380 us; error 33 us; skew 500 ppm
Sep 24 02:59:48 dist-test-slave-sb6p krb5kdc[12109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:48.399458 13143 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 02:59:48.399979 13143 webserver.cc:460] Webserver started at http://127.0.98.1:36435/ using document root <none> and password file <none>
I20260924 02:59:48.400228 13143 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:48.400305 13143 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 02:59:48 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 02:59:48.425670 13143 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:59:48.426491 13189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:48.426685 13143 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 02:59:48.426759 13143 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "19a40e9cd73f476db31d6bb26586bdbf"
format_stamp: "Formatted at 2026-09-24 02:59:47 on dist-test-slave-sb6p"
server_key: "715f5474189abc6c3944aca28f4e50f8"
server_key_iv: "c403c32080a794eb59351618f9b009fb"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:48.427192 13143 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:48.442111 13143 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:48.446836 13143 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:48.447048 13143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:48.447250 13143 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:48.447609 13143 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:59:48.447650 13143 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:48.447680 13143 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:59:48.447717 13143 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:48.453732 13143 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:34079
I20260924 02:59:48.453992 13302 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:34079 every 8 connection(s)
I20260924 02:59:48.454244 13143 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:59:48.460294 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 13143
I20260924 02:59:48.460584 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:46813
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:40093
--webserver_port=40417
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:40309/kms with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 24 02:59:48 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:48.478138 13305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:48.455004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40093 (local address 127.0.98.1:39971)
0924 02:59:48.455250 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:48.455289 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:48.455902 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:48.463450 (+ 7548us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:48.463455 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:48.463745 (+ 290us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:48.464060 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:48.464067 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.465814 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.465815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:48.467180 (+ 1365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:48.467183 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.468030 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.468031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:48.468188 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:48.468745 (+ 557us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:48.468755 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:48.471371 (+ 2616us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:48.476181 (+ 4810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:48.476190 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:48.476192 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:48.476501 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:48.476785 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:48.476786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:48.476786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:48.476922 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:48.477440 (+ 518us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:48.477446 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:48.477614 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 02:59:48.477725 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 02:59:48.477891 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"thread_start_us":85,"threads_started":1}
I20260924 02:59:48.478739 13306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:48.455032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:39971 (local address 127.0.98.62:40093)
0924 02:59:48.462797 (+ 7765us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:48.462807 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:48.462855 (+ 48us) server_negotiation.cc:407] Connection header received
0924 02:59:48.463036 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:48.463044 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:48.463109 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:48.463279 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:48.464203 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.465693 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.467306 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.467923 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.468046 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:48.471507 (+ 3461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:48.471511 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:48.471512 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:48.471525 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:48.476082 (+ 4557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:48.476598 (+ 516us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:48.476602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:48.476602 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:48.476717 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:48.477007 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:48.477008 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:48.477008 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:48.477324 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:48.477410 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:48.478382 (+ 972us) server_negotiation.cc:299] Negotiation successful
0924 02:59:48.478538 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7704,"thread_start_us":70,"threads_started":1}
I20260924 02:59:48.479724 13303 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40093
I20260924 02:59:48.479831 13303 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:48.480029 13303 heartbeater.cc:511] Master 127.0.98.62:40093 requested a full tablet report, sending...
W20260924 02:59:48.480944 13303 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40093 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 02:59:48.481014 13303 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:48.481078 13303 heartbeater.cc:511] Master 127.0.98.62:40093 requested a full tablet report, sending...
I20260924 02:59:48.481473 13303 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:48.481544 13303 heartbeater.cc:511] Master 127.0.98.62:40093 requested a full tablet report, sending...
W20260924 02:59:48.481873 13303 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 02:59:48.572855 13307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:48.573112 13307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:48.573149 13307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:48.574412 13307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:48.574539 13307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:48.574628 13307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:59:48.576152 13307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:40309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:46813
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=40417
--tserver_master_addrs=127.0.98.62:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13307
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:48.576370 13307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:48.576565 13307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:48.581445 13315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:48.581485 13312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:48.581516 13313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:48.581879 13307 server_base.cc:1034] running on GCE node
I20260924 02:59:48.582086 13307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:48.582368 13307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:48.583551 13307 hybrid_clock.cc:648] HybridClock initialized: now 1790218788583524 us; error 39 us; skew 500 ppm
Sep 24 02:59:48 dist-test-slave-sb6p krb5kdc[12109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:48.586566 13307 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 02:59:48.587149 13307 webserver.cc:460] Webserver started at http://127.0.98.2:40417/ using document root <none> and password file <none>
I20260924 02:59:48.587401 13307 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:48.587481 13307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 02:59:48 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 02:59:48.611135 13307 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260924 02:59:48.611831 13323 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:48.612046 13307 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:59:48.612144 13307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "fec5cc6aa5dd4b62af664dc58728d9d1"
format_stamp: "Formatted at 2026-09-24 02:59:47 on dist-test-slave-sb6p"
server_key: "29cb33b098e332efb28a8a7520c0fa2d"
server_key_iv: "1625953806029e2ac9bad02a29c7a051"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:48.612551 13307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:48.627319 13307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:48.629305 13307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:48.629467 13307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:48.629714 13307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:48.630074 13307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:59:48.630128 13307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:48.630164 13307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:59:48.630189 13307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:48.637176 13307 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46813
I20260924 02:59:48.637241 13436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46813 every 8 connection(s)
I20260924 02:59:48.637631 13307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 02:59:48 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:48.645300 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 13307
I20260924 02:59:48.645591 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.98.20:40309/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:32807
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:40093
--webserver_port=44999
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:46067
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:40309/kms with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 02:59:48.650031 13306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:48.638451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42671 (local address 127.0.98.62:40093)
0924 02:59:48.638501 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:48.638502 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:48.639016 (+ 514us) server_negotiation.cc:407] Connection header received
0924 02:59:48.639499 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:48.639500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:48.639513 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:48.639559 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:48.640203 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.640770 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.642281 (+ 1511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.642612 (+ 331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.642659 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:48.646887 (+ 4228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:48.646889 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:48.646889 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:48.646898 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:48.648450 (+ 1552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:48.648940 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:48.648942 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:48.648942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:48.649003 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:48.649295 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:48.649296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:48.649297 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:48.649434 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:48.649503 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:48.649854 (+ 351us) server_negotiation.cc:299] Negotiation successful
0924 02:59:48.649937 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 02:59:48.650240 13439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:48.638464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40093 (local address 127.0.98.2:42671)
0924 02:59:48.638899 (+ 435us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:48.638934 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:48.639409 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:48.639639 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:48.639644 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:48.639857 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:48.640117 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:48.640123 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.640874 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.640875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:48.642181 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:48.642184 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.642760 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.642761 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:48.642895 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:48.643441 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:48.643450 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:48.645040 (+ 1590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:48.648546 (+ 3506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:48.648554 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:48.648556 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:48.648854 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:48.649077 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:48.649079 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:48.649079 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:48.649223 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:48.649520 (+ 297us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:48.649526 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:48.649708 (+ 182us) client_negotiation.cc:769] Sending connection context
0924 02:59:48.649827 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 02:59:48.650005 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":113,"threads_started":1}
I20260924 02:59:48.650847 13437 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40093
I20260924 02:59:48.650951 13437 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:48.651149 13437 heartbeater.cc:511] Master 127.0.98.62:40093 requested a full tablet report, sending...
W20260924 02:59:48.651862 13437 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40093 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 02:59:48.651937 13437 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:48.652004 13437 heartbeater.cc:511] Master 127.0.98.62:40093 requested a full tablet report, sending...
I20260924 02:59:48.652397 13437 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:48.652482 13437 heartbeater.cc:511] Master 127.0.98.62:40093 requested a full tablet report, sending...
W20260924 02:59:48.652860 13437 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 02:59:48.750008 13440 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:48.750346 13440 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:48.750417 13440 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:48.752130 13440 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:48.752221 13440 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:48.752300 13440 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:59:48.754406 13440 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:46067
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.98.20:40309/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:32807
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=44999
--tserver_master_addrs=127.0.98.62:40093
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13440
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:48.754802 13440 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:48.755072 13440 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:48.760574 13446 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:48.760597 13447 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:48.760710 13449 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:48.761551 13440 server_base.cc:1034] running on GCE node
I20260924 02:59:48.761909 13440 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:48.762204 13440 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:48.763427 13440 hybrid_clock.cc:648] HybridClock initialized: now 1790218788763404 us; error 34 us; skew 500 ppm
Sep 24 02:59:48 dist-test-slave-sb6p krb5kdc[12109](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:48.769932 13440 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 02:59:48.770556 13440 webserver.cc:460] Webserver started at http://127.0.98.3:44999/ using document root <none> and password file <none>
I20260924 02:59:48.770817 13440 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:48.770902 13440 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 24 02:59:48 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 02:59:48.784891 13440 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:59:48.785637 13457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:48.785830 13440 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:59:48.785916 13440 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "69fec6b0afa04ace80402ec000dca14d"
format_stamp: "Formatted at 2026-09-24 02:59:47 on dist-test-slave-sb6p"
server_key: "f02be621ca901cf50f9820b454eb2c06"
server_key_iv: "b439d7e67d086128563e4f83ba46ac1e"
server_key_version: "kuduclusterkey@0"
I20260924 02:59:48.786290 13440 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:48.795452 13440 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:48.797107 13440 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:48.797222 13440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:48.797415 13440 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:48.797719 13440 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:59:48.797756 13440 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:48.797784 13440 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:59:48.797804 13440 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:48.804158 13440 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:32807
I20260924 02:59:48.804600 13440 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 02:59:48.805466 13570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:32807 every 8 connection(s)
I20260924 02:59:48.810947 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 13440
Sep 24 02:59:48 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:48.815658 13573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:48.806134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:40093 (local address 127.0.98.3:54567)
0924 02:59:48.806309 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:48.806356 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:48.807023 (+ 667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:48.807267 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:48.807271 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:48.807541 (+ 270us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:48.807819 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:48.807826 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.808463 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.808465 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:48.809790 (+ 1325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:48.809793 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.810138 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.810140 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:48.810255 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:48.810770 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:48.810780 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:48.812351 (+ 1571us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:48.814033 (+ 1682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:48.814040 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:48.814042 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:48.814319 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:48.814542 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:48.814543 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:48.814544 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:48.814689 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:48.814996 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:48.815002 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:48.815171 (+ 169us) client_negotiation.cc:769] Sending connection context
0924 02:59:48.815272 (+ 101us) client_negotiation.cc:240] Negotiation successful
0924 02:59:48.815422 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":87,"threads_started":1}
I20260924 02:59:48.816056 13306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:48.806145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:54567 (local address 127.0.98.62:40093)
0924 02:59:48.806212 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:48.806214 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:48.806448 (+ 234us) server_negotiation.cc:407] Connection header received
0924 02:59:48.807136 (+ 688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:48.807137 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:48.807147 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:48.807175 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:48.807905 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.808376 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.809865 (+ 1489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.810044 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.810092 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:48.812440 (+ 2348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:48.812441 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:48.812442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:48.812449 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:48.813945 (+ 1496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:48.814389 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:48.814390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:48.814391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:48.814454 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:48.814773 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:48.814775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:48.814775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:48.814905 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:48.814967 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:48.815892 (+ 925us) server_negotiation.cc:299] Negotiation successful
0924 02:59:48.815969 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 02:59:48.816509 13571 heartbeater.cc:347] Connected to a master server at 127.0.98.62:40093
I20260924 02:59:48.816848 13571 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:48.817292 13571 heartbeater.cc:511] Master 127.0.98.62:40093 requested a full tablet report, sending...
W20260924 02:59:48.817907 13571 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:40093 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260924 02:59:48.817986 13571 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:48.818063 13571 heartbeater.cc:511] Master 127.0.98.62:40093 requested a full tablet report, sending...
I20260924 02:59:48.818435 13571 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:48.818537 13571 heartbeater.cc:511] Master 127.0.98.62:40093 requested a full tablet report, sending...
W20260924 02:59:48.818894 13571 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 02:59:48.819651 13575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:48.811327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34178 (local address 127.0.98.62:40093)
0924 02:59:48.811431 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:48.811432 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:48.811520 (+ 88us) server_negotiation.cc:407] Connection header received
0924 02:59:48.811576 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:48.811577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:48.811588 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:48.811642 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:48.812489 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.813169 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.814562 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:48.814846 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:48.814909 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:48.816310 (+ 1401us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:48.816312 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:48.816312 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:48.816319 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:48.817860 (+ 1541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:48.818873 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:48.818874 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:48.818875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:48.818935 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:48.819131 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:48.819132 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:48.819132 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:48.819257 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:48.819421 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:48.819477 (+ 56us) server_negotiation.cc:299] Negotiation successful
0924 02:59:48.819558 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":54,"threads_started":1}
Sep 24 02:59:49 dist-test-slave-sb6p krb5kdc[12109](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790218789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:49.482065 13303 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:49.482223 13303 heartbeater.cc:511] Master 127.0.98.62:40093 requested a full tablet report, sending...
I20260924 02:59:49.482589 13105 ts_manager.cc:194] Registered new tserver with Master: 19a40e9cd73f476db31d6bb26586bdbf (127.0.98.1:34079)
I20260924 02:59:49.483579 13105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:39971
Sep 24 02:59:49 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
I20260924 02:59:49.653256 13437 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:49.653507 13437 heartbeater.cc:511] Master 127.0.98.62:40093 requested a full tablet report, sending...
I20260924 02:59:49.653841 13105 ts_manager.cc:194] Registered new tserver with Master: fec5cc6aa5dd4b62af664dc58728d9d1 (127.0.98.2:46813)
I20260924 02:59:49.654578 13105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42671
I20260924 02:59:49.819059 13571 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:49.819205 13571 heartbeater.cc:511] Master 127.0.98.62:40093 requested a full tablet report, sending...
I20260924 02:59:49.819592 13105 ts_manager.cc:194] Registered new tserver with Master: 69fec6b0afa04ace80402ec000dca14d (127.0.98.3:32807)
I20260924 02:59:49.820128 13105 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:54567
I20260924 02:59:49.821529 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 02:59:49.830511 13592 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:49.822553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34194 (local address 127.0.98.62:40093)
0924 02:59:49.822962 (+ 409us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:49.822965 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:49.822971 (+ 6us) server_negotiation.cc:407] Connection header received
0924 02:59:49.823000 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:49.823001 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:49.823012 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:49.823713 (+ 701us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:49.823749 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:49.824469 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:49.825505 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:49.825789 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:49.826120 (+ 331us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:49.827304 (+ 1184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:49.827305 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:49.827306 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:49.827312 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:49.828992 (+ 1680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:49.829415 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:49.829416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:49.829416 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:49.829474 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:49.829664 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:49.829664 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:49.829665 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:49.829784 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:49.830288 (+ 504us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:49.830329 (+ 41us) server_negotiation.cc:299] Negotiation successful
0924 02:59:49.830408 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":59,"threads_started":1}
I20260924 02:59:49.832762 13105 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34194:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 02:59:49.835067 13105 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:59:49.850502 13602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:49.844546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46813 (local address 127.0.0.1:37702)
0924 02:59:49.845584 (+ 1038us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:49.845592 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:49.845630 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:49.845969 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:49.845973 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:49.845976 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:49.846053 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:49.846055 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:49.847564 (+ 1509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:49.847565 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:49.848786 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:49.848789 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:49.850225 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:49.850226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:49.850300 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:49.850313 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 02:59:49.850368 (+ 55us) client_negotiation.cc:240] Negotiation successful
0924 02:59:49.850417 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1026,"thread_start_us":59,"threads_started":1}
I20260924 02:59:49.852347 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:49.844659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37702 (local address 127.0.98.2:46813)
0924 02:59:49.845546 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:49.845551 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:49.845764 (+ 213us) server_negotiation.cc:407] Connection header received
0924 02:59:49.845801 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:49.845803 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:49.845819 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:49.845879 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:49.846122 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:49.847481 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:49.848876 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:49.850060 (+ 1184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:49.850121 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:49.850193 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:49.852226 (+ 2033us) server_negotiation.cc:299] Negotiation successful
0924 02:59:49.852272 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":852,"thread_start_us":62,"threads_started":1}
I20260924 02:59:49.853240 13371 tablet_service.cc:1467] Processing CreateTablet for tablet ff3e519785c14ecd91c70ab89f37a92e (DEFAULT_TABLE table=test-table [id=a08bba1953144f6fa24a6c65c36a7723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:59:49.853626 13371 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3e519785c14ecd91c70ab89f37a92e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:49.855154 13599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:49.844309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:32807 (local address 127.0.0.1:47768)
0924 02:59:49.845285 (+ 976us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:49.845329 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:49.845385 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:49.850871 (+ 5486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:49.850872 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:49.850874 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:49.850914 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:49.850915 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:49.852904 (+ 1989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:49.852905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:49.853982 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:49.853994 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:49.854937 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:49.854938 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:49.855012 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:49.855017 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 02:59:49.855044 (+ 27us) client_negotiation.cc:240] Negotiation successful
0924 02:59:49.855082 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":944,"thread_start_us":78,"threads_started":1}
I20260924 02:59:49.855798 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:49.844847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47768 (local address 127.0.98.3:32807)
0924 02:59:49.849389 (+ 4542us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:49.849393 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:49.849402 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:59:49.849433 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:49.849435 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:49.849450 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:49.850981 (+ 1531us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:49.851065 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:49.852806 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:49.854091 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:49.854844 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:49.855580 (+ 736us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:49.855667 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:49.855690 (+ 23us) server_negotiation.cc:299] Negotiation successful
0924 02:59:49.855730 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4506,"thread_start_us":61,"threads_started":1}
I20260924 02:59:49.856434 13505 tablet_service.cc:1467] Processing CreateTablet for tablet ff3e519785c14ecd91c70ab89f37a92e (DEFAULT_TABLE table=test-table [id=a08bba1953144f6fa24a6c65c36a7723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:59:49.856745 13505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3e519785c14ecd91c70ab89f37a92e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:49.860323 13605 tablet_bootstrap.cc:492] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d: Bootstrap starting.
I20260924 02:59:49.861202 13605 tablet_bootstrap.cc:654] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:49.861558 13605 log.cc:824] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:49.861946 13606 tablet_bootstrap.cc:492] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1: Bootstrap starting.
I20260924 02:59:49.863147 13606 tablet_bootstrap.cc:654] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:49.863575 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:49.844679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34079 (local address 127.0.0.1:45436)
0924 02:59:49.845687 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:49.845694 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:49.845721 (+ 27us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:49.854447 (+ 8726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:49.854449 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:49.854451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:49.854490 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:49.854492 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:49.861240 (+ 6748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:49.861241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:49.862314 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:49.862319 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:49.863353 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:49.863355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:49.863418 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:49.863423 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 02:59:49.863453 (+ 30us) client_negotiation.cc:240] Negotiation successful
0924 02:59:49.863498 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":998,"thread_start_us":47,"threads_started":1}
I20260924 02:59:49.863718 13606 log.cc:824] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:49.864130 13605 tablet_bootstrap.cc:492] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d: No bootstrap required, opened a new log
I20260924 02:59:49.864130 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:49.844779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45436 (local address 127.0.98.1:34079)
0924 02:59:49.854207 (+ 9428us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:49.854212 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:49.854224 (+ 12us) server_negotiation.cc:407] Connection header received
0924 02:59:49.854258 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:49.854260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:49.854275 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:49.854362 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:49.854604 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:49.860026 (+ 5422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:49.862461 (+ 2435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:49.863256 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:49.863899 (+ 643us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:49.863989 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:49.864016 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 02:59:49.864057 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9392,"thread_start_us":47,"threads_started":1}
I20260924 02:59:49.864336 13605 ts_tablet_manager.cc:1397] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 02:59:49.864730 13237 tablet_service.cc:1467] Processing CreateTablet for tablet ff3e519785c14ecd91c70ab89f37a92e (DEFAULT_TABLE table=test-table [id=a08bba1953144f6fa24a6c65c36a7723]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:59:49.865052 13237 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff3e519785c14ecd91c70ab89f37a92e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:49.866914 13605 raft_consensus.cc:348] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fec6b0afa04ace80402ec000dca14d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32807 } } peers { permanent_uuid: "fec5cc6aa5dd4b62af664dc58728d9d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46813 } } peers { permanent_uuid: "19a40e9cd73f476db31d6bb26586bdbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34079 } }
I20260924 02:59:49.867062 13605 raft_consensus.cc:374] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:49.867105 13605 raft_consensus.cc:729] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69fec6b0afa04ace80402ec000dca14d, State: Initialized, Role: FOLLOWER
I20260924 02:59:49.867234 13605 consensus_queue.cc:260] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fec6b0afa04ace80402ec000dca14d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32807 } } peers { permanent_uuid: "fec5cc6aa5dd4b62af664dc58728d9d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46813 } } peers { permanent_uuid: "19a40e9cd73f476db31d6bb26586bdbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34079 } }
I20260924 02:59:49.867498 13605 ts_tablet_manager.cc:1428] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 02:59:49.867604 13571 heartbeater.cc:503] Master 127.0.98.62:40093 was elected leader, sending a full tablet report...
I20260924 02:59:49.869503 13606 tablet_bootstrap.cc:492] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1: No bootstrap required, opened a new log
I20260924 02:59:49.869570 13606 ts_tablet_manager.cc:1397] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20260924 02:59:49.872323 13606 raft_consensus.cc:348] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fec6b0afa04ace80402ec000dca14d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32807 } } peers { permanent_uuid: "fec5cc6aa5dd4b62af664dc58728d9d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46813 } } peers { permanent_uuid: "19a40e9cd73f476db31d6bb26586bdbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34079 } }
I20260924 02:59:49.872512 13606 raft_consensus.cc:374] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:49.872583 13606 raft_consensus.cc:729] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fec5cc6aa5dd4b62af664dc58728d9d1, State: Initialized, Role: FOLLOWER
I20260924 02:59:49.872751 13606 consensus_queue.cc:260] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fec6b0afa04ace80402ec000dca14d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32807 } } peers { permanent_uuid: "fec5cc6aa5dd4b62af664dc58728d9d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46813 } } peers { permanent_uuid: "19a40e9cd73f476db31d6bb26586bdbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34079 } }
I20260924 02:59:49.873028 13606 ts_tablet_manager.cc:1428] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260924 02:59:49.873183 13437 heartbeater.cc:503] Master 127.0.98.62:40093 was elected leader, sending a full tablet report...
I20260924 02:59:49.875269 13609 tablet_bootstrap.cc:492] T ff3e519785c14ecd91c70ab89f37a92e P 19a40e9cd73f476db31d6bb26586bdbf: Bootstrap starting.
I20260924 02:59:49.875914 13609 tablet_bootstrap.cc:654] T ff3e519785c14ecd91c70ab89f37a92e P 19a40e9cd73f476db31d6bb26586bdbf: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:49.876262 13609 log.cc:824] T ff3e519785c14ecd91c70ab89f37a92e P 19a40e9cd73f476db31d6bb26586bdbf: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:49.882752 13609 tablet_bootstrap.cc:492] T ff3e519785c14ecd91c70ab89f37a92e P 19a40e9cd73f476db31d6bb26586bdbf: No bootstrap required, opened a new log
I20260924 02:59:49.882881 13609 ts_tablet_manager.cc:1397] T ff3e519785c14ecd91c70ab89f37a92e P 19a40e9cd73f476db31d6bb26586bdbf: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260924 02:59:49.885589 13609 raft_consensus.cc:348] T ff3e519785c14ecd91c70ab89f37a92e P 19a40e9cd73f476db31d6bb26586bdbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fec6b0afa04ace80402ec000dca14d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32807 } } peers { permanent_uuid: "fec5cc6aa5dd4b62af664dc58728d9d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46813 } } peers { permanent_uuid: "19a40e9cd73f476db31d6bb26586bdbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34079 } }
I20260924 02:59:49.885844 13609 raft_consensus.cc:374] T ff3e519785c14ecd91c70ab89f37a92e P 19a40e9cd73f476db31d6bb26586bdbf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:49.885914 13609 raft_consensus.cc:729] T ff3e519785c14ecd91c70ab89f37a92e P 19a40e9cd73f476db31d6bb26586bdbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19a40e9cd73f476db31d6bb26586bdbf, State: Initialized, Role: FOLLOWER
I20260924 02:59:49.886063 13609 consensus_queue.cc:260] T ff3e519785c14ecd91c70ab89f37a92e P 19a40e9cd73f476db31d6bb26586bdbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fec6b0afa04ace80402ec000dca14d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32807 } } peers { permanent_uuid: "fec5cc6aa5dd4b62af664dc58728d9d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46813 } } peers { permanent_uuid: "19a40e9cd73f476db31d6bb26586bdbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34079 } }
I20260924 02:59:49.886412 13609 ts_tablet_manager.cc:1428] T ff3e519785c14ecd91c70ab89f37a92e P 19a40e9cd73f476db31d6bb26586bdbf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 02:59:49.886579 13303 heartbeater.cc:503] Master 127.0.98.62:40093 was elected leader, sending a full tablet report...
W20260924 02:59:49.888777 13438 tablet.cc:2369] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 24 02:59:49 dist-test-slave-sb6p krb5kdc[12109](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218789, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for HTTP/127.0.98.20@KRBTEST.COM
W20260924 02:59:49.963598 13304 tablet.cc:2369] T ff3e519785c14ecd91c70ab89f37a92e P 19a40e9cd73f476db31d6bb26586bdbf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 02:59:50.057089 13572 tablet.cc:2369] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 02:59:50.097527 13608 raft_consensus.cc:482] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:59:50.097711 13608 raft_consensus.cc:504] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fec6b0afa04ace80402ec000dca14d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32807 } } peers { permanent_uuid: "fec5cc6aa5dd4b62af664dc58728d9d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46813 } } peers { permanent_uuid: "19a40e9cd73f476db31d6bb26586bdbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34079 } }
I20260924 02:59:50.098116 13608 leader_election.cc:288] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fec5cc6aa5dd4b62af664dc58728d9d1 (127.0.98.2:46813), 19a40e9cd73f476db31d6bb26586bdbf (127.0.98.1:34079)
I20260924 02:59:50.111874 13601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:50.098731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:39771 (local address 127.0.98.2:46813)
0924 02:59:50.098790 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:50.098792 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:50.103509 (+ 4717us) server_negotiation.cc:407] Connection header received
0924 02:59:50.108080 (+ 4571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:50.108082 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:50.108096 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:50.108124 (+ 28us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:50.108368 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:50.109134 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:50.110886 (+ 1752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:50.111480 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:50.111525 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:50.111536 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:50.111750 (+ 214us) server_negotiation.cc:299] Negotiation successful
0924 02:59:50.111795 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 02:59:50.112152 13616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:50.098297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46813 (local address 127.0.98.3:39771)
0924 02:59:50.103417 (+ 5120us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:50.103432 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:50.107942 (+ 4510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:50.108196 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:50.108197 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:50.108199 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:50.108290 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:50.108292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:50.109368 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:50.109369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:50.110762 (+ 1393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:50.110763 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:50.111606 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:50.111606 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:50.111672 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:50.111676 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:59:50.112027 (+ 351us) client_negotiation.cc:240] Negotiation successful
0924 02:59:50.112065 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5095,"thread_start_us":83,"threads_started":1}
I20260924 02:59:50.112478 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3e519785c14ecd91c70ab89f37a92e" candidate_uuid: "69fec6b0afa04ace80402ec000dca14d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec5cc6aa5dd4b62af664dc58728d9d1" is_pre_election: true
I20260924 02:59:50.112663 13391 raft_consensus.cc:2454] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69fec6b0afa04ace80402ec000dca14d in term 0.
I20260924 02:59:50.112934 13459 leader_election.cc:302] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69fec6b0afa04ace80402ec000dca14d, fec5cc6aa5dd4b62af664dc58728d9d1; no voters:
I20260924 02:59:50.113081 13608 raft_consensus.cc:2781] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:59:50.113128 13608 raft_consensus.cc:482] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:59:50.113152 13608 raft_consensus.cc:3037] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:50.114020 13608 raft_consensus.cc:504] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fec6b0afa04ace80402ec000dca14d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32807 } } peers { permanent_uuid: "fec5cc6aa5dd4b62af664dc58728d9d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46813 } } peers { permanent_uuid: "19a40e9cd73f476db31d6bb26586bdbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34079 } }
I20260924 02:59:50.114151 13608 leader_election.cc:288] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [CANDIDATE]: Term 1 election: Requested vote from peers fec5cc6aa5dd4b62af664dc58728d9d1 (127.0.98.2:46813), 19a40e9cd73f476db31d6bb26586bdbf (127.0.98.1:34079)
I20260924 02:59:50.114332 13391 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3e519785c14ecd91c70ab89f37a92e" candidate_uuid: "69fec6b0afa04ace80402ec000dca14d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fec5cc6aa5dd4b62af664dc58728d9d1"
I20260924 02:59:50.114408 13391 raft_consensus.cc:3037] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:50.114713 13603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:50.098787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55509 (local address 127.0.98.1:34079)
0924 02:59:50.098845 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:50.098847 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:50.110665 (+ 11818us) server_negotiation.cc:407] Connection header received
0924 02:59:50.110760 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:50.110761 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:50.110775 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:50.110804 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:50.110991 (+ 187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:50.111665 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:50.113536 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:50.114066 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:50.114113 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:50.114123 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:50.114600 (+ 477us) server_negotiation.cc:299] Negotiation successful
0924 02:59:50.114649 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 02:59:50.115013 13617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:50.098485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34079 (local address 127.0.98.3:55509)
0924 02:59:50.110594 (+ 12109us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:50.110607 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:50.110692 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:50.110865 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:50.110866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:50.110868 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:50.110922 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:50.110923 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:50.111745 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:50.111746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:50.113372 (+ 1626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:50.113375 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:50.114278 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:50.114279 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:50.114345 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:50.114348 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:59:50.114899 (+ 551us) client_negotiation.cc:240] Negotiation successful
0924 02:59:50.114941 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12087,"thread_start_us":96,"threads_started":1}
I20260924 02:59:50.115397 13391 raft_consensus.cc:2454] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69fec6b0afa04ace80402ec000dca14d in term 1.
I20260924 02:59:50.115619 13459 leader_election.cc:302] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69fec6b0afa04ace80402ec000dca14d, fec5cc6aa5dd4b62af664dc58728d9d1; no voters:
I20260924 02:59:50.115850 13608 raft_consensus.cc:2781] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:59:50.115916 13257 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3e519785c14ecd91c70ab89f37a92e" candidate_uuid: "69fec6b0afa04ace80402ec000dca14d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a40e9cd73f476db31d6bb26586bdbf" is_pre_election: true
I20260924 02:59:50.116040 13608 raft_consensus.cc:686] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [term 1 LEADER]: Becoming Leader. State: Replica: 69fec6b0afa04ace80402ec000dca14d, State: Running, Role: LEADER
I20260924 02:59:50.116097 13257 raft_consensus.cc:2454] T ff3e519785c14ecd91c70ab89f37a92e P 19a40e9cd73f476db31d6bb26586bdbf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69fec6b0afa04ace80402ec000dca14d in term 0.
I20260924 02:59:50.116142 13608 consensus_queue.cc:237] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fec6b0afa04ace80402ec000dca14d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32807 } } peers { permanent_uuid: "fec5cc6aa5dd4b62af664dc58728d9d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46813 } } peers { permanent_uuid: "19a40e9cd73f476db31d6bb26586bdbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34079 } }
I20260924 02:59:50.116215 13256 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff3e519785c14ecd91c70ab89f37a92e" candidate_uuid: "69fec6b0afa04ace80402ec000dca14d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "19a40e9cd73f476db31d6bb26586bdbf"
I20260924 02:59:50.116269 13256 raft_consensus.cc:3037] T ff3e519785c14ecd91c70ab89f37a92e P 19a40e9cd73f476db31d6bb26586bdbf [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:50.116935 13105 catalog_manager.cc:5495] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d reported cstate change: term changed from 0 to 1, leader changed from <none> to 69fec6b0afa04ace80402ec000dca14d (127.0.98.3). New cstate: current_term: 1 leader_uuid: "69fec6b0afa04ace80402ec000dca14d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69fec6b0afa04ace80402ec000dca14d" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 32807 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fec5cc6aa5dd4b62af664dc58728d9d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46813 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "19a40e9cd73f476db31d6bb26586bdbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34079 } health_report { overall_health: UNKNOWN } } }
I20260924 02:59:50.117064 13256 raft_consensus.cc:2454] T ff3e519785c14ecd91c70ab89f37a92e P 19a40e9cd73f476db31d6bb26586bdbf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69fec6b0afa04ace80402ec000dca14d in term 1.
I20260924 02:59:50.133172 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:50.129853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47776 (local address 127.0.98.3:32807)
0924 02:59:50.129965 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:50.129967 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:50.129974 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:59:50.130019 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:50.130020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:50.130041 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:50.130112 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:59:50.130294 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:50.131053 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:50.132070 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:50.132370 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:50.132432 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:50.132669 (+ 237us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:59:50.132802 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:59:50.132870 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:50.133070 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 02:59:50.133091 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260924 02:59:50.139772 13256 raft_consensus.cc:1261] T ff3e519785c14ecd91c70ab89f37a92e P 19a40e9cd73f476db31d6bb26586bdbf [term 1 FOLLOWER]: Refusing update from remote peer 69fec6b0afa04ace80402ec000dca14d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:59:50.139791 13391 raft_consensus.cc:1261] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1 [term 1 FOLLOWER]: Refusing update from remote peer 69fec6b0afa04ace80402ec000dca14d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:59:50.140208 13618 consensus_queue.cc:1035] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [LEADER]: Connected to new peer: Peer: permanent_uuid: "fec5cc6aa5dd4b62af664dc58728d9d1" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46813 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:59:50.142104 13618 consensus_queue.cc:1035] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [LEADER]: Connected to new peer: Peer: permanent_uuid: "19a40e9cd73f476db31d6bb26586bdbf" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 34079 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:59:50.145931 13621 mvcc.cc:205] Tried to move back new op lower bound from 7332736164408745984 to 7332736164316241920. Current Snapshot: MvccSnapshot[applied={T|T < 7332736164408745984}]
I20260924 02:59:50.146075 13622 mvcc.cc:205] Tried to move back new op lower bound from 7332736164408745984 to 7332736164316241920. Current Snapshot: MvccSnapshot[applied={T|T < 7332736164408745984}]
I20260924 02:59:50.148375 13624 mvcc.cc:205] Tried to move back new op lower bound from 7332736164408745984 to 7332736164316241920. Current Snapshot: MvccSnapshot[applied={T|T < 7332736164408745984}]
I20260924 02:59:50.151522 13105 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:59:50.151628 13105 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34194:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:59:50.152294 13105 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 79d6a3f118b74dafb186cfe9b9e99bf1: Sending DeleteTablet for 3 replicas of tablet ff3e519785c14ecd91c70ab89f37a92e
I20260924 02:59:50.152818 13505 tablet_service.cc:1514] Processing DeleteTablet for tablet ff3e519785c14ecd91c70ab89f37a92e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:59:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:47768
I20260924 02:59:50.153045 13371 tablet_service.cc:1514] Processing DeleteTablet for tablet ff3e519785c14ecd91c70ab89f37a92e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:59:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37702
I20260924 02:59:50.153258 13237 tablet_service.cc:1514] Processing DeleteTablet for tablet ff3e519785c14ecd91c70ab89f37a92e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:59:50 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45436
I20260924 02:59:50.154042 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 13143
I20260924 02:59:50.158056 13631 tablet_replica.cc:330] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1: stopping tablet replica
I20260924 02:59:50.158181 13631 raft_consensus.cc:2229] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:59:50.158306 13631 raft_consensus.cc:2258] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:59:50.161999 13630 tablet_replica.cc:330] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d: stopping tablet replica
I20260924 02:59:50.162120 13630 raft_consensus.cc:2229] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [term 1 LEADER]: Raft consensus shutting down.
I20260924 02:59:50.162257 13630 raft_consensus.cc:2258] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:59:50.162896 13631 ts_tablet_manager.cc:1905] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:59:50.164892 13630 ts_tablet_manager.cc:1905] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:59:50.166396 13631 ts_tablet_manager.cc:1918] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:59:50.166483 13631 log.cc:1196] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/ff3e519785c14ecd91c70ab89f37a92e
I20260924 02:59:50.166769 13631 ts_tablet_manager.cc:1939] T ff3e519785c14ecd91c70ab89f37a92e P fec5cc6aa5dd4b62af664dc58728d9d1: Deleting consensus metadata
I20260924 02:59:50.167273 13090 catalog_manager.cc:4841] TS fec5cc6aa5dd4b62af664dc58728d9d1 (127.0.98.2:46813): tablet ff3e519785c14ecd91c70ab89f37a92e (table test-table [id=a08bba1953144f6fa24a6c65c36a7723]) successfully deleted
I20260924 02:59:50.167480 13630 ts_tablet_manager.cc:1918] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:59:50.167554 13630 log.cc:1196] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/ff3e519785c14ecd91c70ab89f37a92e
I20260924 02:59:50.167790 13630 ts_tablet_manager.cc:1939] T ff3e519785c14ecd91c70ab89f37a92e P 69fec6b0afa04ace80402ec000dca14d: Deleting consensus metadata
I20260924 02:59:50.168231 13089 catalog_manager.cc:4841] TS 69fec6b0afa04ace80402ec000dca14d (127.0.98.3:32807): tablet ff3e519785c14ecd91c70ab89f37a92e (table test-table [id=a08bba1953144f6fa24a6c65c36a7723]) successfully deleted
W20260924 02:59:50.168480 13090 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:34079 (error 108)
I20260924 02:59:50.169341 13600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:50.169224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:34079 (local address 127.0.0.1:45442)
0924 02:59:50.169275 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:50.169307 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34079: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 02:59:50.169456 13090 catalog_manager.cc:4584] TS 19a40e9cd73f476db31d6bb26586bdbf (127.0.98.1:34079): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ff3e519785c14ecd91c70ab89f37a92e: Network error: Client connection negotiation failed: client connection to 127.0.98.1:34079: connect: Connection refused (error 111)
I20260924 02:59:50.169647 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 13307
I20260924 02:59:50.177114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 13440
I20260924 02:59:50.183897 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 13072
I20260924 02:59:50.191875 392 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260924 02:59:50.525573 392 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260924 02:59:50.525667 392 mini_ranger.cc:66] Stopping Ranger...
I20260924 02:59:50.859333 392 mini_ranger.cc:68] Stopped Ranger
2026-09-24 02:59:50.859 UTC [12139] LOG: received smart shutdown request
2026-09-24 02:59:50.863 UTC [12139] LOG: background worker "logical replication launcher" (PID 12148) exited with exit code 1
2026-09-24 02:59:50.863 UTC [12143] LOG: shutting down
2026-09-24 02:59:51.011 UTC [12139] LOG: database system is shut down
2026-09-24T02:59:51Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45807 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:59:51 dist-test-slave-sb6p krb5kdc[13642](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:59:51 dist-test-slave-sb6p krb5kdc[13642](info): set up 2 sockets
Sep 24 02:59:51 dist-test-slave-sb6p krb5kdc[13642](info): commencing operation
krb5kdc: starting...
W20260924 02:59:52.046111 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:59:52 dist-test-slave-sb6p krb5kdc[13642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:59:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:59:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 02:59:52.155483 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34879
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34879,127.0.98.61:38915,127.0.98.60:39777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:52.232522 13655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:52.232720 13655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:52.232740 13655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:52.233855 13655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:59:52.233892 13655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:52.233906 13655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:52.233917 13655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:59:52.233928 13655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:59:52.235409 13655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34879,127.0.98.61:38915,127.0.98.60:39777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13655
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:52.235630 13655 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:52.235837 13655 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 02:59:52.240422 13655 server_base.cc:1034] running on GCE node
W20260924 02:59:52.240530 13663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:52.240448 13660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:52.240434 13661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:52.240831 13655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:52.241101 13655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:52.242246 13655 hybrid_clock.cc:648] HybridClock initialized: now 1790218792242231 us; error 28 us; skew 500 ppm
Sep 24 02:59:52 dist-test-slave-sb6p krb5kdc[13642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:52.244585 13655 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:59:52.244946 13655 webserver.cc:460] Webserver started at http://127.0.98.62:38313/ using document root <none> and password file <none>
I20260924 02:59:52.245124 13655 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:52.245163 13655 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:52.245244 13655 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:59:52.246206 13655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "9c12efd504104148b1d445a7ab4d4921"
format_stamp: "Formatted at 2026-09-24 02:59:52 on dist-test-slave-sb6p"
server_key: "76d24fcd1e99d4d1c50e74507567d51f"
server_key_iv: "b4c39dccc43f8c4755dd95817c87ee8a"
server_key_version: "encryptionkey@0"
I20260924 02:59:52.246493 13655 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "9c12efd504104148b1d445a7ab4d4921"
format_stamp: "Formatted at 2026-09-24 02:59:52 on dist-test-slave-sb6p"
server_key: "76d24fcd1e99d4d1c50e74507567d51f"
server_key_iv: "b4c39dccc43f8c4755dd95817c87ee8a"
server_key_version: "encryptionkey@0"
I20260924 02:59:52.247691 13655 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:59:52.248307 13670 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:52.248459 13655 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:52.248538 13655 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "9c12efd504104148b1d445a7ab4d4921"
format_stamp: "Formatted at 2026-09-24 02:59:52 on dist-test-slave-sb6p"
server_key: "76d24fcd1e99d4d1c50e74507567d51f"
server_key_iv: "b4c39dccc43f8c4755dd95817c87ee8a"
server_key_version: "encryptionkey@0"
I20260924 02:59:52.248590 13655 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:52.264279 13655 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:52.265955 13655 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:52.266060 13655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:52.269681 13655 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34879
I20260924 02:59:52.269718 13722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34879 every 8 connection(s)
I20260924 02:59:52.270010 13655 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:59:52.270880 13723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:52.272361 13723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 02:59:52 dist-test-slave-sb6p krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:52.279635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 13655
I20260924 02:59:52.279716 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:59:52.279903 392 external_mini_cluster.cc:1428] Setting key 5cf865e734b3fefbef245e7a5f4dff35
I20260924 02:59:52.281261 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.272997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39560 (local address 127.0.98.62:34879)
0924 02:59:52.273144 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:52.273153 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:52.273234 (+ 81us) server_negotiation.cc:407] Connection header received
0924 02:59:52.273715 (+ 481us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:52.273716 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:52.273733 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:52.273792 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:52.274277 (+ 485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.275098 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.275907 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.276201 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.276269 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.278249 (+ 1980us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:52.278256 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:52.278256 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:52.278263 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:52.279693 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.280163 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.280165 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.280166 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.280242 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.280466 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.280467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.280467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.280654 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:52.280709 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:52.280891 (+ 182us) server_negotiation.cc:299] Negotiation successful
0924 02:59:52.281011 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":17,"server-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260924 02:59:52.281268 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.272974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34879 (local address 127.0.0.1:39560)
0924 02:59:52.273123 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.273159 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:52.273586 (+ 427us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:52.273811 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:52.273813 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:52.274042 (+ 229us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:52.274215 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.274219 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.275182 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.275182 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.275845 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.275847 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.276290 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.276291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.276354 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.276857 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:52.276865 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:52.278171 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:52.279794 (+ 1623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.279801 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.279804 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.280077 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.280302 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.280303 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.280303 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.280409 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.280705 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:52.280710 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:52.280782 (+ 72us) client_negotiation.cc:769] Sending connection context
0924 02:59:52.280875 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 02:59:52.281087 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":62,"thread_start_us":60,"threads_started":1}
I20260924 02:59:52.282037 13723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } has no permanent_uuid. Determining permanent_uuid...
I20260924 02:59:52.282356 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.282256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38915 (local address 127.0.0.1:55548)
0924 02:59:52.282300 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.282337 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 02:59:52.282487 13671 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38915: connect: Connection refused (error 111)
I20260924 02:59:52.282908 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.282827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38915 (local address 127.0.0.1:55556)
0924 02:59:52.282868 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.282891 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 02:59:52.283006 13723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38915: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38915: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 02:59:52.301694 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38915
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34879,127.0.98.61:38915,127.0.98.60:39777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 02:59:52.332216 13723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } attempt: 1
I20260924 02:59:52.332702 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.332541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38915 (local address 127.0.0.1:55558)
0924 02:59:52.332608 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.332662 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260924 02:59:52.333030 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.332941 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38915 (local address 127.0.0.1:55562)
0924 02:59:52.332985 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.333011 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260924 02:59:52.333123 13723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38915: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38915: connect: Connection refused (error 111)
W20260924 02:59:52.378983 13730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:52.379173 13730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:52.379194 13730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:52.380283 13730 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:59:52.380322 13730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:52.380336 13730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:52.380358 13730 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:59:52.380374 13730 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:59:52.381830 13730 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34879,127.0.98.61:38915,127.0.98.60:39777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:38915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13730
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:52.382047 13730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:52.382242 13730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:52.386700 13738 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:52.386719 13736 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:52.386813 13730 server_base.cc:1034] running on GCE node
W20260924 02:59:52.386722 13735 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:52.387094 13730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:52.387374 13730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:52.388510 13730 hybrid_clock.cc:648] HybridClock initialized: now 1790218792388500 us; error 26 us; skew 500 ppm
Sep 24 02:59:52 dist-test-slave-sb6p krb5kdc[13642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:52.390713 13730 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 02:59:52.391098 13730 webserver.cc:460] Webserver started at http://127.0.98.61:38729/ using document root <none> and password file <none>
I20260924 02:59:52.391294 13730 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:52.391330 13730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:52.391415 13730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:59:52.392341 13730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-1/data/instance:
uuid: "4dc317e410f44b6cbfadbd753dd738aa"
format_stamp: "Formatted at 2026-09-24 02:59:52 on dist-test-slave-sb6p"
server_key: "ebd836798de12d262ccfa0e37bbb4208"
server_key_iv: "5251d880d828cc889e4f0d5eb8b33c8e"
server_key_version: "encryptionkey@0"
I20260924 02:59:52.392611 13730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-1/wal/instance:
uuid: "4dc317e410f44b6cbfadbd753dd738aa"
format_stamp: "Formatted at 2026-09-24 02:59:52 on dist-test-slave-sb6p"
server_key: "ebd836798de12d262ccfa0e37bbb4208"
server_key_iv: "5251d880d828cc889e4f0d5eb8b33c8e"
server_key_version: "encryptionkey@0"
I20260924 02:59:52.393679 13730 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:59:52.394260 13745 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:52.394405 13730 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:52.394479 13730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-1/wal
uuid: "4dc317e410f44b6cbfadbd753dd738aa"
format_stamp: "Formatted at 2026-09-24 02:59:52 on dist-test-slave-sb6p"
server_key: "ebd836798de12d262ccfa0e37bbb4208"
server_key_iv: "5251d880d828cc889e4f0d5eb8b33c8e"
server_key_version: "encryptionkey@0"
I20260924 02:59:52.394554 13730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:52.401264 13723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } attempt: 2
I20260924 02:59:52.401623 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.401496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38915 (local address 127.0.0.1:55574)
0924 02:59:52.401550 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.401587 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 02:59:52.401911 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.401841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38915 (local address 127.0.0.1:55586)
0924 02:59:52.401874 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.401895 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38915: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 02:59:52.401986 13723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:38915: Network error: Client connection negotiation failed: client connection to 127.0.98.61:38915: connect: Connection refused (error 111)
I20260924 02:59:52.412386 13730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:52.414036 13730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:52.414153 13730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:52.417732 13730 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:38915
I20260924 02:59:52.417761 13797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:38915 every 8 connection(s)
I20260924 02:59:52.418058 13730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-1/data/info.pb
I20260924 02:59:52.418874 13798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:52.420331 13798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 02:59:52 dist-test-slave-sb6p krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:52.425612 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 13730
I20260924 02:59:52.425732 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-1/wal/instance
I20260924 02:59:52.425915 392 external_mini_cluster.cc:1428] Setting key c1f21c53a7cb070c06e58ac951916822
I20260924 02:59:52.428375 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.420960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39572 (local address 127.0.98.62:34879)
0924 02:59:52.421028 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:52.421031 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:52.421190 (+ 159us) server_negotiation.cc:407] Connection header received
0924 02:59:52.421620 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:52.421622 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:52.421633 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:52.421671 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:52.422203 (+ 532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.422747 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.423638 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.423826 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.423866 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.425666 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:52.425668 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:52.425668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:52.425675 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:52.427058 (+ 1383us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.427477 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.427479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.427479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.427544 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.427746 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.427747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.427747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.427880 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:52.427943 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:52.428205 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 02:59:52.428287 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260924 02:59:52.428529 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.420931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34879 (local address 127.0.0.1:39572)
0924 02:59:52.421069 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.421103 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:52.421538 (+ 435us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:52.421724 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:52.421728 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:52.421955 (+ 227us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:52.422145 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.422149 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.422805 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.422806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.423577 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.423579 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.423873 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.423874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.423970 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.424355 (+ 385us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:52.424363 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:52.425584 (+ 1221us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:52.427150 (+ 1566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.427156 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.427157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.427402 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.427602 (+ 200us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.427603 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.427603 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.427695 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.427948 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:52.427955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:52.428085 (+ 130us) client_negotiation.cc:769] Sending connection context
0924 02:59:52.428178 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 02:59:52.428324 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":66,"threads_started":1}
I20260924 02:59:52.429029 13798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 02:59:52 dist-test-slave-sb6p krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 02:59:52.438311 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.429314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38915 (local address 127.0.0.1:55590)
0924 02:59:52.429360 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.429370 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:52.429410 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:52.429661 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:52.429662 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:52.429765 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:52.429810 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.429811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.431017 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.431018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.431489 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.431491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.432075 (+ 584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.432076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.432134 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.432467 (+ 333us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:52.432471 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:52.435699 (+ 3228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:52.437297 (+ 1598us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.437298 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.437299 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.437510 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.437683 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.437684 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.437684 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.437739 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.438001 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:52.438002 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:52.438039 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 02:59:52.438099 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 02:59:52.438198 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 02:59:52.438369 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.429371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55590 (local address 127.0.98.61:38915)
0924 02:59:52.429510 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:52.429515 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:52.429537 (+ 22us) server_negotiation.cc:407] Connection header received
0924 02:59:52.429556 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:52.429558 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:52.429571 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:52.429653 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:52.429870 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.430953 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.431559 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.432008 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.432083 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.435773 (+ 3690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:52.435780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:52.435781 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:52.435787 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:52.437226 (+ 1439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.437584 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.437586 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.437587 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.437632 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.437790 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.437791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.437791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.437941 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:52.438012 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:52.438136 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 02:59:52.438247 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":50,"server-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260924 02:59:52.438946 13798 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } has no permanent_uuid. Determining permanent_uuid...
I20260924 02:59:52.439244 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.439141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39777 (local address 127.0.0.1:41964)
0924 02:59:52.439179 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.439222 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 02:59:52.439360 13746 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39777: connect: Connection refused (error 111)
I20260924 02:59:52.439752 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.439676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39777 (local address 127.0.0.1:41978)
0924 02:59:52.439709 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.439731 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 02:59:52.439851 13798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39777: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39777: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 02:59:52.449012 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39777
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:32841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34879,127.0.98.61:38915,127.0.98.60:39777
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 02:59:52.489123 13798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } attempt: 1
I20260924 02:59:52.489558 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.489442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39777 (local address 127.0.0.1:41994)
0924 02:59:52.489493 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.489526 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 02:59:52.489833 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.489756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39777 (local address 127.0.0.1:42010)
0924 02:59:52.489790 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.489811 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 02:59:52.489959 13798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39777: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39777: connect: Connection refused (error 111)
I20260924 02:59:52.493134 13723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } attempt: 3
Sep 24 02:59:52 dist-test-slave-sb6p krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 02:59:52.499738 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.493385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38915 (local address 127.0.0.1:55596)
0924 02:59:52.493432 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.493442 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:52.493480 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:52.493714 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:52.493715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:52.493861 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:52.493922 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.493924 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.494575 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.494576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.495084 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.495085 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.495433 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.495434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.495508 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.495947 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:52.495952 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:52.497324 (+ 1372us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:52.498835 (+ 1511us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.498837 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.498837 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.499001 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.499167 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.499168 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.499168 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.499226 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.499430 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:52.499431 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:52.499486 (+ 55us) client_negotiation.cc:769] Sending connection context
0924 02:59:52.499543 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 02:59:52.499642 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 02:59:52.499737 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.493541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55596 (local address 127.0.98.61:38915)
0924 02:59:52.493587 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:52.493589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:52.493596 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:59:52.493640 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:52.493640 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:52.493649 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:52.493693 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:52.493989 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.494482 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.495160 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.495376 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.495429 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.497402 (+ 1973us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:52.497404 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:52.497404 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:52.497411 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:52.498770 (+ 1359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.499062 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.499063 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.499063 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.499115 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.499274 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.499274 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.499275 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.499378 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:52.499433 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:52.499569 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 02:59:52.499644 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 02:59:52.500093 13723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } has no permanent_uuid. Determining permanent_uuid...
I20260924 02:59:52.500398 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.500315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39777 (local address 127.0.0.1:42026)
0924 02:59:52.500349 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.500378 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 02:59:52.500644 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.500585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39777 (local address 127.0.0.1:42034)
0924 02:59:52.500610 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.500630 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 02:59:52.500715 13723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39777: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39777: connect: Connection refused (error 111)
W20260924 02:59:52.529078 13805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:52.529279 13805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:52.529307 13805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:52.530408 13805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:59:52.530454 13805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:52.530473 13805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:52.530488 13805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:59:52.530506 13805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:59:52.531847 13723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } attempt: 1
I20260924 02:59:52.532048 13805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:32841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34879,127.0.98.61:38915,127.0.98.60:39777
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:39777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:52.532214 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.532104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39777 (local address 127.0.0.1:42050)
0924 02:59:52.532157 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.532188 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 02:59:52.532294 13805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:52.532506 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.532409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39777 (local address 127.0.0.1:42054)
0924 02:59:52.532464 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.532487 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260924 02:59:52.532538 13805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:52.532588 13723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39777: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39777: connect: Connection refused (error 111)
W20260924 02:59:52.537036 13813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:52.537045 13811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:52.537123 13810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:52.537268 13805 server_base.cc:1034] running on GCE node
I20260924 02:59:52.537536 13805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:52.537792 13805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:52.538934 13805 hybrid_clock.cc:648] HybridClock initialized: now 1790218792538918 us; error 26 us; skew 500 ppm
Sep 24 02:59:52 dist-test-slave-sb6p krb5kdc[13642](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:52.541214 13805 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 02:59:52.541666 13805 webserver.cc:460] Webserver started at http://127.0.98.60:34095/ using document root <none> and password file <none>
I20260924 02:59:52.541853 13805 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:52.541899 13805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:52.542021 13805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:59:52.543066 13805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-2/data/instance:
uuid: "35baa906892547ef92b24668b58467e8"
format_stamp: "Formatted at 2026-09-24 02:59:52 on dist-test-slave-sb6p"
server_key: "1f0c7b687b9af48ad5f70cb68b8ca251"
server_key_iv: "a013842d9f26dd68f2251a669fcd9140"
server_key_version: "encryptionkey@0"
I20260924 02:59:52.543377 13805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-2/wal/instance:
uuid: "35baa906892547ef92b24668b58467e8"
format_stamp: "Formatted at 2026-09-24 02:59:52 on dist-test-slave-sb6p"
server_key: "1f0c7b687b9af48ad5f70cb68b8ca251"
server_key_iv: "a013842d9f26dd68f2251a669fcd9140"
server_key_version: "encryptionkey@0"
I20260924 02:59:52.544523 13805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:59:52.545138 13820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:52.545315 13805 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:59:52.545398 13805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-2/wal
uuid: "35baa906892547ef92b24668b58467e8"
format_stamp: "Formatted at 2026-09-24 02:59:52 on dist-test-slave-sb6p"
server_key: "1f0c7b687b9af48ad5f70cb68b8ca251"
server_key_iv: "a013842d9f26dd68f2251a669fcd9140"
server_key_version: "encryptionkey@0"
I20260924 02:59:52.545456 13805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:52.558087 13798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } attempt: 2
I20260924 02:59:52.558446 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.558345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39777 (local address 127.0.0.1:42056)
0924 02:59:52.558392 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.558420 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 02:59:52.558763 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.558683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39777 (local address 127.0.0.1:42062)
0924 02:59:52.558720 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.558743 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39777: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 02:59:52.558838 13798 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:39777: Network error: Client connection negotiation failed: client connection to 127.0.98.60:39777: connect: Connection refused (error 111)
I20260924 02:59:52.562914 13805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:52.564872 13805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:52.564998 13805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:52.568687 13805 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:39777
I20260924 02:59:52.568755 13872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:39777 every 8 connection(s)
I20260924 02:59:52.569037 13805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-2/data/info.pb
I20260924 02:59:52.569787 13873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:52.571255 13873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } has no permanent_uuid. Determining permanent_uuid...
I20260924 02:59:52.573016 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 13805
I20260924 02:59:52.573101 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790218647334776-392-0/minicluster-data/master-2/wal/instance
I20260924 02:59:52.573297 392 external_mini_cluster.cc:1428] Setting key 3526514251b0dea0ffdd269ca1a6887b
Sep 24 02:59:52 dist-test-slave-sb6p krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 02:59:52 dist-test-slave-sb6p krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 02:59:52.579852 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.571849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39588 (local address 127.0.98.62:34879)
0924 02:59:52.571898 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:52.571900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:52.572048 (+ 148us) server_negotiation.cc:407] Connection header received
0924 02:59:52.572494 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:52.572495 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:52.572507 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:52.572556 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:52.573142 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.573696 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.574655 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.574955 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.575024 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.577189 (+ 2165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:52.577190 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:52.577191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:52.577197 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:52.578456 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.578876 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.578878 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.578878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.578934 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.579208 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.579209 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.579209 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.579337 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:52.579400 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:52.579683 (+ 283us) server_negotiation.cc:299] Negotiation successful
0924 02:59:52.579763 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 02:59:52.580006 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.571813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:34879 (local address 127.0.0.1:39588)
0924 02:59:52.571942 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.571973 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:52.572404 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:52.572636 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:52.572640 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:52.572868 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:52.573050 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.573054 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.573790 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.573791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.574584 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.574586 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.575021 (+ 435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.575022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.575136 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.575578 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:52.575587 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:52.577105 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:52.578549 (+ 1444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.578556 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.578558 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.578807 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.579001 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.579001 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.579002 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.579146 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.579397 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:52.579402 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:52.579553 (+ 151us) client_negotiation.cc:769] Sending connection context
0924 02:59:52.579664 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 02:59:52.579800 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":59,"threads_started":1}
I20260924 02:59:52.580538 13873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } has no permanent_uuid. Determining permanent_uuid...
I20260924 02:59:52.582002 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.573603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42066 (local address 127.0.98.60:39777)
0924 02:59:52.573752 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:52.573756 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:52.573778 (+ 22us) server_negotiation.cc:407] Connection header received
0924 02:59:52.573836 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:52.573839 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:52.573857 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:52.573915 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:52.574452 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.575407 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.576332 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.576770 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.576838 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.579020 (+ 2182us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:52.579027 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:52.579027 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:52.579033 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:52.580798 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.581175 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.581178 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.581179 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.581246 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.581416 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.581417 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.581417 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.581635 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:52.581798 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:52.581837 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 02:59:52.581907 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
Sep 24 02:59:52 dist-test-slave-sb6p krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 02:59:52 dist-test-slave-sb6p krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:52.586862 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.580758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:38915 (local address 127.0.0.1:55612)
0924 02:59:52.580800 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.580809 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:52.580841 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:52.580945 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:52.580946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:52.581063 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:52.581094 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.581095 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.582013 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.582014 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.582672 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.582674 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.583085 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.583085 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.583158 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.583509 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:52.583514 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:52.584571 (+ 1057us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:52.586005 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.586005 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.586006 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.586150 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.586316 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.586317 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.586317 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.586356 (+ 39us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.586594 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:52.586594 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:52.586634 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 02:59:52.586694 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 02:59:52.586773 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 02:59:52.586870 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.580813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55612 (local address 127.0.98.61:38915)
0924 02:59:52.580859 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:52.580861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:52.580870 (+ 9us) server_negotiation.cc:407] Connection header received
0924 02:59:52.580892 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:52.580893 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:52.580901 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:52.580940 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:52.581186 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.581861 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.582735 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.583021 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.583072 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.584647 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:52.584648 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:52.584648 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:52.584655 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:52.585929 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.586209 (+ 280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.586210 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.586210 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.586263 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.586415 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.586416 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.586417 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.586543 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:52.586601 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:52.586727 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 02:59:52.586788 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 02:59:52.587172 13873 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } has no permanent_uuid. Determining permanent_uuid...
I20260924 02:59:52.588953 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.582732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39590 (local address 127.0.98.62:34879)
0924 02:59:52.582792 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:52.582794 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:52.582819 (+ 25us) server_negotiation.cc:407] Connection header received
0924 02:59:52.582862 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:52.582863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:52.582872 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:52.582928 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:52.583411 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.583870 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.584521 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.584749 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.584809 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.586767 (+ 1958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:52.586768 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:52.586768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:52.586774 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:52.588039 (+ 1265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.588312 (+ 273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.588313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.588313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.588362 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.588511 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.588512 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.588512 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.588608 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:52.588774 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:52.588808 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 02:59:52.588871 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
Sep 24 02:59:52 dist-test-slave-sb6p krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 02:59:52.592741 13875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.587392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39777 (local address 127.0.0.1:42080)
0924 02:59:52.587441 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.587450 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:52.587506 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:52.587625 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:52.587626 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:52.587736 (+ 110us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:52.587758 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.587759 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.588307 (+ 548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.588308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.588751 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.588752 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.589065 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.589066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.589126 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.589445 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:52.589450 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:52.590425 (+ 975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:52.591926 (+ 1501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.591927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.591928 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.592067 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.592231 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.592232 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.592232 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.592281 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.592466 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:52.592467 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:52.592500 (+ 33us) client_negotiation.cc:769] Sending connection context
0924 02:59:52.592558 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 02:59:52.592647 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 02:59:52.592792 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.587482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42080 (local address 127.0.98.60:39777)
0924 02:59:52.587510 (+ 28us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:52.587512 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:52.587519 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:59:52.587555 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:52.587556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:52.587566 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:52.587609 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:52.587813 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.588248 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.588811 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.589008 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.589064 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.590490 (+ 1426us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:52.590491 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:52.590491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:52.590534 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:52.591861 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.592139 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.592140 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.592140 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.592184 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.592328 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.592329 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.592329 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.592419 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:52.592469 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:52.592595 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 02:59:52.592693 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":13}
Sep 24 02:59:52 dist-test-slave-sb6p krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218792, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 02:59:52.594210 13873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8: Bootstrap starting.
I20260924 02:59:52.595199 13873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:52.595480 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.589480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55622 (local address 127.0.98.61:38915)
0924 02:59:52.589512 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:52.589513 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:52.589520 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:59:52.589557 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:52.589558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:52.589566 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:52.589607 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:52.589982 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.590476 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.591222 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.591389 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.591436 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.593271 (+ 1835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:52.593272 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:52.593272 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:52.593278 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:52.594574 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.594856 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.594857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.594857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.594912 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.595034 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.595034 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.595035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.595143 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:52.595291 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:52.595330 (+ 39us) server_negotiation.cc:299] Negotiation successful
0924 02:59:52.595398 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 02:59:52.595503 13873 log.cc:824] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:52.596191 13873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8: No bootstrap required, opened a new log
I20260924 02:59:52.596935 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 02:59:52.597815 13873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } }
I20260924 02:59:52.597939 13873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:52.597961 13873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35baa906892547ef92b24668b58467e8, State: Initialized, Role: FOLLOWER
I20260924 02:59:52.598074 13873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } }
I20260924 02:59:52.598371 13873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:59:52.598347 13884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } } }
I20260924 02:59:52.598461 13884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 02:59:52.600301 13805 master_runner.cc:186] Error getting master registration for 127.0.98.62:34879: OK, instance_id { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" instance_seqno: 1790218792268691 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 02:59:52.602980 13805 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260924 02:59:52.603417 13897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 02:59:52.603525 13897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 02:59:52.605955 13726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.598108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39598 (local address 127.0.98.62:34879)
0924 02:59:52.598151 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:52.598153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:52.598260 (+ 107us) server_negotiation.cc:407] Connection header received
0924 02:59:52.598309 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:52.598310 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:52.598320 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:52.598347 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:52.599185 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.599841 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.600762 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.601055 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.601104 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.602426 (+ 1322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:52.602427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:52.602427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:52.602434 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:52.604029 (+ 1595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.604337 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.604338 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.604339 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.604395 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.604593 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.604594 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.604594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.604917 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:52.605449 (+ 532us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:52.605796 (+ 347us) server_negotiation.cc:299] Negotiation successful
0924 02:59:52.605869 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 02:59:52.606395 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.598737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42082 (local address 127.0.98.60:39777)
0924 02:59:52.599650 (+ 913us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:52.599652 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:52.599660 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:59:52.599676 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:52.599677 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:52.599684 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:52.600551 (+ 867us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:52.600576 (+ 25us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.601215 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.602098 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.602384 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.602675 (+ 291us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.603592 (+ 917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:52.603593 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:52.603594 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:52.603599 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:52.605039 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.605406 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.605407 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.605408 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.605458 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.605658 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.605659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.605659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.605790 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:52.606208 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:52.606245 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 02:59:52.606316 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":898}
I20260924 02:59:52.606485 13802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.598713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55628 (local address 127.0.98.61:38915)
0924 02:59:52.598746 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:52.598748 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:52.598815 (+ 67us) server_negotiation.cc:407] Connection header received
0924 02:59:52.598871 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:52.598872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:52.598881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:52.598938 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:52.599511 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.600134 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.601050 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.601318 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.601616 (+ 298us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.603695 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:52.603696 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:52.603696 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:52.603702 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:52.605356 (+ 1654us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.605706 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.605707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.605707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.605761 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.605934 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.605935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.605935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.606046 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:52.606128 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:52.606330 (+ 202us) server_negotiation.cc:299] Negotiation successful
0924 02:59:52.606397 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 02:59:52.607725 13723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } attempt: 2
Sep 24 02:59:52 dist-test-slave-sb6p krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 02:59:52.613900 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.608123 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42090 (local address 127.0.98.60:39777)
0924 02:59:52.608169 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:52.608170 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:52.608176 (+ 6us) server_negotiation.cc:407] Connection header received
0924 02:59:52.608190 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:52.608191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:52.608197 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:52.608238 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:52.608512 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.608934 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.609813 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.609988 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.610044 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.611849 (+ 1805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:52.611850 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:52.611851 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:52.611857 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:52.612998 (+ 1141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.613280 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.613281 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.613281 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.613328 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.613479 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.613479 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.613480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.613573 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:52.613631 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:52.613754 (+ 123us) server_negotiation.cc:299] Negotiation successful
0924 02:59:52.613816 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 02:59:52.613914 13725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.607936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39777 (local address 127.0.0.1:42090)
0924 02:59:52.607987 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.607999 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:52.608041 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:52.608262 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:52.608263 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:52.608410 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:52.608448 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.608450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.608997 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.608998 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.609746 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.609748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.610042 (+ 294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.610042 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.610095 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.610437 (+ 342us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:52.610442 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:52.611783 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:52.613069 (+ 1286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.613069 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.613070 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.613224 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.613378 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.613379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.613379 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.613430 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.613622 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:52.613623 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:52.613671 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 02:59:52.613729 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 02:59:52.613815 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 02:59:52.615255 13723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921: Bootstrap starting.
I20260924 02:59:52.615862 13723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:52.616101 13723 log.cc:824] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:52.616801 13723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921: No bootstrap required, opened a new log
I20260924 02:59:52.618295 13723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } }
I20260924 02:59:52.618399 13723 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:52.618419 13723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c12efd504104148b1d445a7ab4d4921, State: Initialized, Role: FOLLOWER
I20260924 02:59:52.618562 13723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } }
I20260924 02:59:52.618793 13900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } } }
I20260924 02:59:52.618847 13723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:59:52.618870 13900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 02:59:52.620401 13911 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 02:59:52.620481 13911 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 02:59:52.621111 13655 master_runner.cc:186] Error getting master registration for 127.0.98.61:38915: OK, instance_id { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" instance_seqno: 1790218792416742 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 02:59:52.623903 13655 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 02:59:52.649974 13798 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } attempt: 3
Sep 24 02:59:52 dist-test-slave-sb6p krb5kdc[13642](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 02:59:52.657878 13877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.650565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42098 (local address 127.0.98.60:39777)
0924 02:59:52.650630 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:52.650631 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:52.650641 (+ 10us) server_negotiation.cc:407] Connection header received
0924 02:59:52.650664 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:52.650665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:52.650675 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:52.650734 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:52.651058 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.651826 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.652792 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.653089 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.653159 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.655267 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:52.655268 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:52.655269 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:52.655276 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:52.656791 (+ 1515us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.657151 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.657152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.657152 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.657210 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:52.657383 (+ 173us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:52.657384 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:52.657384 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:52.657507 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:52.657567 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:52.657710 (+ 143us) server_negotiation.cc:299] Negotiation successful
0924 02:59:52.657783 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 02:59:52.657878 13800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:52.650428 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:39777 (local address 127.0.0.1:42098)
0924 02:59:52.650540 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:52.650555 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:52.650593 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:52.650747 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:52.650748 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:52.650932 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:52.650982 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.650984 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.651897 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.651898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.652714 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:52.652716 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:52.653156 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:52.653157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:52.653215 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:52.653677 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:52.653682 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:52.655172 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:52.656870 (+ 1698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.656871 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.656871 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.657077 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.657267 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:52.657268 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:52.657269 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:52.657329 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:52.657565 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:52.657566 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:52.657612 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 02:59:52.657679 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 02:59:52.657775 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260924 02:59:52.659730 13798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa: Bootstrap starting.
I20260924 02:59:52.660459 13798 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:52.660768 13798 log.cc:824] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:52.661629 13798 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa: No bootstrap required, opened a new log
I20260924 02:59:52.664150 13798 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } }
I20260924 02:59:52.664286 13798 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:52.664318 13798 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4dc317e410f44b6cbfadbd753dd738aa, State: Initialized, Role: FOLLOWER
I20260924 02:59:52.664484 13798 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } }
I20260924 02:59:52.664788 13798 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:59:52.664862 13914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } } }
I20260924 02:59:52.664987 13914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa [sys.catalog]: This master's current role is: FOLLOWER
W20260924 02:59:52.666962 13925 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 02:59:52.667070 13925 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 02:59:52.669032 13730 master_runner.cc:263] No leader master found from master 4dc317e410f44b6cbfadbd753dd738aa
I20260924 02:59:52.669075 13730 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 02:59:52.741504 13884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:59:52.741621 13884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } }
I20260924 02:59:52.741846 13884 leader_election.cc:288] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c12efd504104148b1d445a7ab4d4921 (127.0.98.62:34879), 4dc317e410f44b6cbfadbd753dd738aa (127.0.98.61:38915)
I20260924 02:59:52.742479 13697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35baa906892547ef92b24668b58467e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c12efd504104148b1d445a7ab4d4921" is_pre_election: true
I20260924 02:59:52.742667 13697 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35baa906892547ef92b24668b58467e8 in term 0.
I20260924 02:59:52.742857 13772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35baa906892547ef92b24668b58467e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc317e410f44b6cbfadbd753dd738aa" is_pre_election: true
I20260924 02:59:52.742992 13772 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35baa906892547ef92b24668b58467e8 in term 0.
I20260924 02:59:52.743294 13821 leader_election.cc:302] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35baa906892547ef92b24668b58467e8, 4dc317e410f44b6cbfadbd753dd738aa; no voters:
I20260924 02:59:52.743650 13884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:59:52.743709 13884 raft_consensus.cc:482] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:59:52.743731 13884 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:52.744560 13884 raft_consensus.cc:504] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } }
I20260924 02:59:52.744680 13884 leader_election.cc:288] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [CANDIDATE]: Term 1 election: Requested vote from peers 9c12efd504104148b1d445a7ab4d4921 (127.0.98.62:34879), 4dc317e410f44b6cbfadbd753dd738aa (127.0.98.61:38915)
I20260924 02:59:52.745921 13772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35baa906892547ef92b24668b58467e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4dc317e410f44b6cbfadbd753dd738aa"
I20260924 02:59:52.745993 13772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:52.746623 13697 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "35baa906892547ef92b24668b58467e8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c12efd504104148b1d445a7ab4d4921"
I20260924 02:59:52.746699 13697 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:52.747519 13697 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35baa906892547ef92b24668b58467e8 in term 1.
I20260924 02:59:52.747715 13823 leader_election.cc:302] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 35baa906892547ef92b24668b58467e8, 9c12efd504104148b1d445a7ab4d4921; no voters:
I20260924 02:59:52.748139 13772 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35baa906892547ef92b24668b58467e8 in term 1.
I20260924 02:59:52.748477 13884 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:59:52.748631 13884 raft_consensus.cc:686] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [term 1 LEADER]: Becoming Leader. State: Replica: 35baa906892547ef92b24668b58467e8, State: Running, Role: LEADER
I20260924 02:59:52.748728 13884 consensus_queue.cc:237] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } }
I20260924 02:59:52.749054 13884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35baa906892547ef92b24668b58467e8. Latest consensus state: current_term: 1 leader_uuid: "35baa906892547ef92b24668b58467e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } } }
I20260924 02:59:52.749106 13884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [sys.catalog]: This master's current role is: LEADER
I20260924 02:59:52.751727 13929 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:59:52.754745 13929 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:59:52.756728 13697 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921 [term 1 FOLLOWER]: Refusing update from remote peer 35baa906892547ef92b24668b58467e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:59:52.757069 13884 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:59:52.758350 13900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35baa906892547ef92b24668b58467e8. Latest consensus state: current_term: 1 leader_uuid: "35baa906892547ef92b24668b58467e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } } }
I20260924 02:59:52.758441 13900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 02:59:52.758639 13772 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa [term 1 FOLLOWER]: Refusing update from remote peer 35baa906892547ef92b24668b58467e8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:59:52.758888 13884 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:59:52.759960 13914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader 35baa906892547ef92b24668b58467e8. Latest consensus state: current_term: 1 leader_uuid: "35baa906892547ef92b24668b58467e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } } }
I20260924 02:59:52.760043 13914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa [sys.catalog]: This master's current role is: FOLLOWER
I20260924 02:59:52.763454 13927 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35baa906892547ef92b24668b58467e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } } }
I20260924 02:59:52.763540 13927 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [sys.catalog]: This master's current role is: LEADER
I20260924 02:59:52.763609 13930 mvcc.cc:205] Tried to move back new op lower bound from 7332736175128887296 to 7332736175099297792. Current Snapshot: MvccSnapshot[applied={T|T < 7332736175128887296}]
I20260924 02:59:52.763608 13914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35baa906892547ef92b24668b58467e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } } }
I20260924 02:59:52.763695 13914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa [sys.catalog]: This master's current role is: FOLLOWER
I20260924 02:59:52.763983 13933 mvcc.cc:205] Tried to move back new op lower bound from 7332736175128887296 to 7332736175099297792. Current Snapshot: MvccSnapshot[applied={T|T < 7332736175128887296}]
W20260924 02:59:52.766709 13676 tablet.cc:2369] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 02:59:52.767275 13884 sys_catalog.cc:455] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35baa906892547ef92b24668b58467e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } } }
I20260924 02:59:52.767360 13884 sys_catalog.cc:458] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8 [sys.catalog]: This master's current role is: LEADER
I20260924 02:59:52.767481 13929 catalog_manager.cc:1347] Generated new cluster ID: a64c187b4fec4918aaac9d7024a3b421
I20260924 02:59:52.767541 13929 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:59:52.767776 13900 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35baa906892547ef92b24668b58467e8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c12efd504104148b1d445a7ab4d4921" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34879 } } peers { permanent_uuid: "4dc317e410f44b6cbfadbd753dd738aa" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 38915 } } peers { permanent_uuid: "35baa906892547ef92b24668b58467e8" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 39777 } } }
I20260924 02:59:52.767853 13900 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 02:59:52.787268 13929 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:59:52.788003 13929 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:59:52.794090 13929 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 35baa906892547ef92b24668b58467e8: Generated new TSK 0
I20260924 02:59:52.794257 13929 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 02:59:53.620903 13911 catalog_manager.cc:1259] Loaded cluster ID: a64c187b4fec4918aaac9d7024a3b421
I20260924 02:59:53.620978 13911 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921: loading cluster ID for follower catalog manager: success
I20260924 02:59:53.621804 13911 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921: acquiring CA information for follower catalog manager: success
I20260924 02:59:53.622089 13911 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9c12efd504104148b1d445a7ab4d4921: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 02:59:53.667538 13925 catalog_manager.cc:1259] Loaded cluster ID: a64c187b4fec4918aaac9d7024a3b421
I20260924 02:59:53.667609 13925 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa: loading cluster ID for follower catalog manager: success
I20260924 02:59:53.668408 13925 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa: acquiring CA information for follower catalog manager: success
I20260924 02:59:53.668700 13925 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 4dc317e410f44b6cbfadbd753dd738aa: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 02:59:53.839660 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 13655
I20260924 02:59:53.844242 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 13730
I20260924 02:59:53.848278 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 13805
2026-09-24T02:59:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2748 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:59:53 dist-test-slave-sb6p krb5kdc[13944](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:59:53 dist-test-slave-sb6p krb5kdc[13944](info): set up 2 sockets
Sep 24 02:59:53 dist-test-slave-sb6p krb5kdc[13944](info): commencing operation
krb5kdc: starting...
W20260924 02:59:54.790302 392 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:59:54 dist-test-slave-sb6p krb5kdc[13944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:59:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:59:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 02:59:54.905541 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37665
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:54.983683 13957 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:54.983876 13957 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:54.983897 13957 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:54.985024 13957 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:59:54.985065 13957 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:54.985078 13957 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:54.985091 13957 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:59:54.985116 13957 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:59:54.986567 13957 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13957
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:54.986783 13957 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:54.986979 13957 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:54.991503 13962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:54.991523 13965 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:54.991552 13957 server_base.cc:1034] running on GCE node
W20260924 02:59:54.991609 13963 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:54.991932 13957 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:54.992188 13957 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:54.993346 13957 hybrid_clock.cc:648] HybridClock initialized: now 1790218794993322 us; error 38 us; skew 500 ppm
Sep 24 02:59:54 dist-test-slave-sb6p krb5kdc[13944](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:54.995698 13957 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:59:54.996068 13957 webserver.cc:460] Webserver started at http://127.0.98.62:39809/ using document root <none> and password file <none>
I20260924 02:59:54.996273 13957 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:54.996320 13957 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:54.996436 13957 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:59:54.997416 13957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "2486a9ee844343518e52c6ab12dd37c4"
format_stamp: "Formatted at 2026-09-24 02:59:54 on dist-test-slave-sb6p"
server_key: "39c9da89150a7aae3a8b42f259e48854"
server_key_iv: "0ef1929889fda3ccdb75ed6d5bf6c4d0"
server_key_version: "encryptionkey@0"
I20260924 02:59:54.997741 13957 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "2486a9ee844343518e52c6ab12dd37c4"
format_stamp: "Formatted at 2026-09-24 02:59:54 on dist-test-slave-sb6p"
server_key: "39c9da89150a7aae3a8b42f259e48854"
server_key_iv: "0ef1929889fda3ccdb75ed6d5bf6c4d0"
server_key_version: "encryptionkey@0"
I20260924 02:59:54.998921 13957 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 02:59:54.999526 13972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:54.999672 13957 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:54.999742 13957 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "2486a9ee844343518e52c6ab12dd37c4"
format_stamp: "Formatted at 2026-09-24 02:59:54 on dist-test-slave-sb6p"
server_key: "39c9da89150a7aae3a8b42f259e48854"
server_key_iv: "0ef1929889fda3ccdb75ed6d5bf6c4d0"
server_key_version: "encryptionkey@0"
I20260924 02:59:54.999799 13957 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:55.009132 13957 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:55.011051 13957 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:55.011184 13957 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:55.014784 13957 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37665
I20260924 02:59:55.014827 14024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37665 every 8 connection(s)
I20260924 02:59:55.015126 13957 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:59:55.015888 14025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:55.018043 14025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4: Bootstrap starting.
I20260924 02:59:55.018597 14025 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:55.018850 14025 log.cc:824] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:55.019500 14025 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4: No bootstrap required, opened a new log
I20260924 02:59:55.019752 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 13957
I20260924 02:59:55.019842 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:59:55.020013 392 external_mini_cluster.cc:1428] Setting key 13e3f0a33f20508410a168d873cea27e
I20260924 02:59:55.021394 14025 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2486a9ee844343518e52c6ab12dd37c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37665 } }
I20260924 02:59:55.021524 14025 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:55.021556 14025 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2486a9ee844343518e52c6ab12dd37c4, State: Initialized, Role: FOLLOWER
I20260924 02:59:55.021705 14025 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2486a9ee844343518e52c6ab12dd37c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37665 } }
I20260924 02:59:55.021780 14025 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:59:55.021811 14025 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:59:55.021857 14025 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:55.022704 14025 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2486a9ee844343518e52c6ab12dd37c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37665 } }
I20260924 02:59:55.022795 14025 leader_election.cc:302] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2486a9ee844343518e52c6ab12dd37c4; no voters:
I20260924 02:59:55.022938 14025 leader_election.cc:288] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:59:55.022969 14030 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:59:55.023067 14030 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [term 1 LEADER]: Becoming Leader. State: Replica: 2486a9ee844343518e52c6ab12dd37c4, State: Running, Role: LEADER
I20260924 02:59:55.023180 14025 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:59:55.023211 14030 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2486a9ee844343518e52c6ab12dd37c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37665 } }
I20260924 02:59:55.023530 14030 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2486a9ee844343518e52c6ab12dd37c4. Latest consensus state: current_term: 1 leader_uuid: "2486a9ee844343518e52c6ab12dd37c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2486a9ee844343518e52c6ab12dd37c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37665 } } }
I20260924 02:59:55.023721 14030 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [sys.catalog]: This master's current role is: LEADER
I20260924 02:59:55.023660 14031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2486a9ee844343518e52c6ab12dd37c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2486a9ee844343518e52c6ab12dd37c4" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37665 } } }
I20260924 02:59:55.023936 14031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4 [sys.catalog]: This master's current role is: LEADER
I20260924 02:59:55.024086 14038 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:59:55.024586 14038 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:59:55.026290 14038 catalog_manager.cc:1347] Generated new cluster ID: 8d341a94156e448cb6e87a3de071a692
I20260924 02:59:55.026374 14038 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 02:59:55 dist-test-slave-sb6p krb5kdc[13944](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218794, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:55.030509 14029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:55.020505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51264 (local address 127.0.98.62:37665)
0924 02:59:55.020681 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:55.020690 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:55.020713 (+ 23us) server_negotiation.cc:407] Connection header received
0924 02:59:55.021365 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:55.021370 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:55.021414 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:55.021583 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:55.022228 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:55.023334 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:55.024345 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:55.024943 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:55.025059 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:55.027533 (+ 2474us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:55.027537 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:55.027537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:55.027548 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:55.029030 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:55.029348 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:55.029351 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:55.029352 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:55.029459 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:55.029629 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:55.029630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:55.029630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:55.029852 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:55.030008 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:55.030073 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 02:59:55.030204 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260924 02:59:55.031585 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 02:59:55.033177 14038 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:59:55.033813 14038 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:59:55.039232 14038 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2486a9ee844343518e52c6ab12dd37c4: Generated new TSK 0
I20260924 02:59:55.039381 14038 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 02:59:55.085048 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 13957
2026-09-24T02:59:55Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1235 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 02:59:55 dist-test-slave-sb6p krb5kdc[14052](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 02:59:55 dist-test-slave-sb6p krb5kdc[14052](info): set up 2 sockets
Sep 24 02:59:55 dist-test-slave-sb6p krb5kdc[14052](info): commencing operation
krb5kdc: starting...
W20260924 02:59:56.034267 392 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 02:59:56 dist-test-slave-sb6p krb5kdc[14052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T02:59:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T02:59:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 02:59:56.142403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37233
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:56.219532 14065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:56.219740 14065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:56.219770 14065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:56.220918 14065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:59:56.220960 14065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:56.220978 14065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:56.220994 14065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:59:56.221009 14065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:59:56.222457 14065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14065
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:56.222718 14065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:56.222921 14065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:56.227829 14071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:56.227947 14065 server_base.cc:1034] running on GCE node
W20260924 02:59:56.227829 14070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:56.227852 14073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:56.228327 14065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:56.228588 14065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:56.229724 14065 hybrid_clock.cc:648] HybridClock initialized: now 1790218796229709 us; error 28 us; skew 500 ppm
Sep 24 02:59:56 dist-test-slave-sb6p krb5kdc[14052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:56.232182 14065 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:59:56.232563 14065 webserver.cc:460] Webserver started at http://127.0.98.62:38659/ using document root <none> and password file <none>
I20260924 02:59:56.232744 14065 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:56.232784 14065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:56.232868 14065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:59:56.233834 14065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "4b4cbd6845cc487ea0f61fc513b04362"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "ea037272ac09a2779f9a15a263d777cf"
server_key_iv: "72eb0e889d191d5585121b749c8f3ff1"
server_key_version: "encryptionkey@0"
I20260924 02:59:56.234109 14065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "4b4cbd6845cc487ea0f61fc513b04362"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "ea037272ac09a2779f9a15a263d777cf"
server_key_iv: "72eb0e889d191d5585121b749c8f3ff1"
server_key_version: "encryptionkey@0"
I20260924 02:59:56.235306 14065 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 02:59:56.235911 14080 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:56.236057 14065 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:59:56.236132 14065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "4b4cbd6845cc487ea0f61fc513b04362"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "ea037272ac09a2779f9a15a263d777cf"
server_key_iv: "72eb0e889d191d5585121b749c8f3ff1"
server_key_version: "encryptionkey@0"
I20260924 02:59:56.236202 14065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:56.267828 14065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:56.269542 14065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:56.269646 14065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:56.273252 14065 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37233
I20260924 02:59:56.273296 14132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37233 every 8 connection(s)
I20260924 02:59:56.273583 14065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:59:56.274501 14133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:56.276599 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 14065
I20260924 02:59:56.276685 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 02:59:56.276831 14133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362: Bootstrap starting.
I20260924 02:59:56.276922 392 external_mini_cluster.cc:1428] Setting key c02958588623885db5b03f8849fd5de5
I20260924 02:59:56.277449 14133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:56.277766 14133 log.cc:824] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:56.278381 14133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362: No bootstrap required, opened a new log
I20260924 02:59:56.280452 14133 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } }
I20260924 02:59:56.280620 14133 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:56.280660 14133 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4cbd6845cc487ea0f61fc513b04362, State: Initialized, Role: FOLLOWER
I20260924 02:59:56.280778 14133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } }
I20260924 02:59:56.280843 14133 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:59:56.280877 14133 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:59:56.280920 14133 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:56.281428 14133 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } }
I20260924 02:59:56.281522 14133 leader_election.cc:302] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b4cbd6845cc487ea0f61fc513b04362; no voters:
I20260924 02:59:56.281682 14133 leader_election.cc:288] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 02:59:56.281711 14138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:59:56.281872 14138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 1 LEADER]: Becoming Leader. State: Replica: 4b4cbd6845cc487ea0f61fc513b04362, State: Running, Role: LEADER
I20260924 02:59:56.281924 14133 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:59:56.282006 14138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } }
I20260924 02:59:56.282325 14138 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b4cbd6845cc487ea0f61fc513b04362. Latest consensus state: current_term: 1 leader_uuid: "4b4cbd6845cc487ea0f61fc513b04362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } } }
I20260924 02:59:56.282548 14138 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [sys.catalog]: This master's current role is: LEADER
I20260924 02:59:56.282461 14140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b4cbd6845cc487ea0f61fc513b04362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } } }
I20260924 02:59:56.282900 14140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [sys.catalog]: This master's current role is: LEADER
I20260924 02:59:56.283320 14147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 24 02:59:56 dist-test-slave-sb6p krb5kdc[14052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218796, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:56.284056 14147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:59:56.286509 14147 catalog_manager.cc:1347] Generated new cluster ID: cba6ce4fd0a24cebb0419f12450fd020
I20260924 02:59:56.286741 14147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:59:56.287744 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.277447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53982 (local address 127.0.98.62:37233)
0924 02:59:56.277702 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:56.277709 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:56.277736 (+ 27us) server_negotiation.cc:407] Connection header received
0924 02:59:56.278211 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:56.278218 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:56.278267 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:56.278383 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:56.279131 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.280004 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.281036 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.281446 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.281556 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.284304 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:56.284308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:56.284308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:56.284319 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:56.286133 (+ 1814us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:56.286500 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:56.286503 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:56.286503 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:56.286630 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:56.286832 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:56.286833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:56.286833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:56.287119 (+ 286us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:56.287255 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:56.287392 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 02:59:56.287544 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":79,"threads_started":1}
I20260924 02:59:56.296808 14147 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 02:59:56.297287 14147 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:59:56.303834 14147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362: Generated new TSK 0
I20260924 02:59:56.304018 14147 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 02:59:56.333776 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37233
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:56.413789 14159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:56.414006 14159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:56.414028 14159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:56.415180 14159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:56.415220 14159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:56.415246 14159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:59:56.416635 14159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:56.416816 14159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:56.416986 14159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:56.421644 14167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:56.421648 14164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:56.421896 14165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:56.421819 14159 server_base.cc:1034] running on GCE node
I20260924 02:59:56.422154 14159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:56.422381 14159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:56.423511 14159 hybrid_clock.cc:648] HybridClock initialized: now 1790218796423484 us; error 37 us; skew 500 ppm
Sep 24 02:59:56 dist-test-slave-sb6p krb5kdc[14052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:56.425803 14159 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 02:59:56.426199 14159 webserver.cc:460] Webserver started at http://127.0.98.1:38223/ using document root <none> and password file <none>
I20260924 02:59:56.426388 14159 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:56.426434 14159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:56.426581 14159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:59:56.427569 14159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "fa85f532f07f4f89bd994bc9b8849e47"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "69d51b78ab3bcfecbde7066c2c8f1bb4"
server_key_iv: "5bfe85a8cfe30143fff7a5f8f2af49ff"
server_key_version: "encryptionkey@0"
I20260924 02:59:56.427889 14159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "fa85f532f07f4f89bd994bc9b8849e47"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "69d51b78ab3bcfecbde7066c2c8f1bb4"
server_key_iv: "5bfe85a8cfe30143fff7a5f8f2af49ff"
server_key_version: "encryptionkey@0"
I20260924 02:59:56.429036 14159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 02:59:56.429657 14174 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:56.429807 14159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:59:56.429893 14159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "fa85f532f07f4f89bd994bc9b8849e47"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "69d51b78ab3bcfecbde7066c2c8f1bb4"
server_key_iv: "5bfe85a8cfe30143fff7a5f8f2af49ff"
server_key_version: "encryptionkey@0"
I20260924 02:59:56.429955 14159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:56.438686 14159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:56.440408 14159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:56.440538 14159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:56.440733 14159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:56.441000 14159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:59:56.441036 14159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:56.441064 14159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:59:56.441088 14159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:56.446271 14159 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37619
I20260924 02:59:56.446321 14287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37619 every 8 connection(s)
I20260924 02:59:56.446640 14159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:59:56.448134 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 14159
I20260924 02:59:56.448235 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 02:59:56.448405 392 external_mini_cluster.cc:1428] Setting key 43ff31528111e5c697cd2c4606a5319e
Sep 24 02:59:56 dist-test-slave-sb6p krb5kdc[14052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 02:59:56.455497 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.447315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57935 (local address 127.0.98.62:37233)
0924 02:59:56.447362 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:56.447364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:56.447583 (+ 219us) server_negotiation.cc:407] Connection header received
0924 02:59:56.448077 (+ 494us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:56.448078 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:56.448090 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:56.448139 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:56.449154 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.449641 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.450594 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.450809 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.450862 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.453021 (+ 2159us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:56.453022 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:56.453022 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:56.453029 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:56.454254 (+ 1225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:56.454625 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:56.454626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:56.454627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:56.454683 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:56.454890 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:56.454890 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:56.454891 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:56.455003 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:56.455073 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:56.455321 (+ 248us) server_negotiation.cc:299] Negotiation successful
0924 02:59:56.455406 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 02:59:56.455668 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.447298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:57935)
0924 02:59:56.447473 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:56.447507 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:56.447976 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:56.448211 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:56.448215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:56.448840 (+ 625us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:56.449075 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:56.449080 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.449721 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.449722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:56.450496 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:56.450498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.450876 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.450877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:56.450999 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.451471 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:56.451479 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:56.452934 (+ 1455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:56.454324 (+ 1390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:56.454331 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:56.454333 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:56.454570 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:56.454734 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:56.454735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:56.454735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:56.454839 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:56.455053 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:56.455060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:56.455206 (+ 146us) client_negotiation.cc:769] Sending connection context
0924 02:59:56.455295 (+ 89us) client_negotiation.cc:240] Negotiation successful
0924 02:59:56.455448 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":90,"threads_started":1}
I20260924 02:59:56.456123 14288 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37233
I20260924 02:59:56.456212 14288 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:56.456396 14288 heartbeater.cc:511] Master 127.0.98.62:37233 requested a full tablet report, sending...
I20260924 02:59:56.456871 14097 ts_manager.cc:194] Registered new tserver with Master: fa85f532f07f4f89bd994bc9b8849e47 (127.0.98.1:37619)
I20260924 02:59:56.457540 14097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57935
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 02:59:56.472786 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37233
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:56.552542 14293 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:56.552742 14293 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:56.552763 14293 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:56.553892 14293 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:56.553931 14293 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:56.553958 14293 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:59:56.555374 14293 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14293
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:56.555586 14293 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:56.555783 14293 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:56.560436 14299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:56.560443 14301 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:56.560551 14298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:56.560596 14293 server_base.cc:1034] running on GCE node
I20260924 02:59:56.560850 14293 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:56.561069 14293 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:56.562204 14293 hybrid_clock.cc:648] HybridClock initialized: now 1790218796562191 us; error 27 us; skew 500 ppm
Sep 24 02:59:56 dist-test-slave-sb6p krb5kdc[14052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:56.564385 14293 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 02:59:56.564766 14293 webserver.cc:460] Webserver started at http://127.0.98.2:39783/ using document root <none> and password file <none>
I20260924 02:59:56.564947 14293 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:56.564992 14293 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:56.565109 14293 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:59:56.566072 14293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "003c7eeafd1646848d258201048b8147"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "285d2b73fa1958fb253067b0850c305f"
server_key_iv: "01cbd91d4321f72dce97091b8cee5577"
server_key_version: "encryptionkey@0"
I20260924 02:59:56.566394 14293 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "003c7eeafd1646848d258201048b8147"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "285d2b73fa1958fb253067b0850c305f"
server_key_iv: "01cbd91d4321f72dce97091b8cee5577"
server_key_version: "encryptionkey@0"
I20260924 02:59:56.567615 14293 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 02:59:56.568225 14308 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:56.568377 14293 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 02:59:56.568454 14293 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "003c7eeafd1646848d258201048b8147"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "285d2b73fa1958fb253067b0850c305f"
server_key_iv: "01cbd91d4321f72dce97091b8cee5577"
server_key_version: "encryptionkey@0"
I20260924 02:59:56.568512 14293 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:56.601776 14293 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:56.603575 14293 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:56.603708 14293 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:56.603930 14293 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:56.604225 14293 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:59:56.604259 14293 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:56.604287 14293 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:59:56.604308 14293 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:56.609650 14293 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39559
I20260924 02:59:56.609709 14421 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39559 every 8 connection(s)
I20260924 02:59:56.609992 14293 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 02:59:56 dist-test-slave-sb6p krb5kdc[14052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:56.617322 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 14293
I20260924 02:59:56.617419 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 02:59:56.617583 392 external_mini_cluster.cc:1428] Setting key 02770159d03372d10f1a4d9aaf261a75
I20260924 02:59:56.618589 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.610605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:49153 (local address 127.0.98.62:37233)
0924 02:59:56.610657 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:56.610659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:56.610861 (+ 202us) server_negotiation.cc:407] Connection header received
0924 02:59:56.611357 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:56.611359 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:56.611371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:56.611417 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:56.612312 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.612813 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.613758 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.613948 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.614002 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.615969 (+ 1967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:56.615970 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:56.615971 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:56.615978 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:56.617218 (+ 1240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:56.617622 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:56.617624 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:56.617624 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:56.617700 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:56.617928 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:56.617929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:56.617929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:56.618034 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:56.618092 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:56.618390 (+ 298us) server_negotiation.cc:299] Negotiation successful
0924 02:59:56.618471 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 02:59:56.618745 14424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.610613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:49153)
0924 02:59:56.610745 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:56.610781 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:56.611263 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:56.611498 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:56.611502 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:56.612033 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:56.612219 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:56.612223 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.612892 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.612893 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:56.613678 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:56.613680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.614015 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.614016 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:56.614135 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.614592 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:56.614600 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:56.615884 (+ 1284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:56.617303 (+ 1419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:56.617310 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:56.617311 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:56.617559 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:56.617756 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:56.617757 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:56.617757 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:56.617881 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:56.618085 (+ 204us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:56.618090 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:56.618258 (+ 168us) client_negotiation.cc:769] Sending connection context
0924 02:59:56.618358 (+ 100us) client_negotiation.cc:240] Negotiation successful
0924 02:59:56.618535 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
I20260924 02:59:56.619140 14422 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37233
I20260924 02:59:56.619222 14422 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:56.619385 14422 heartbeater.cc:511] Master 127.0.98.62:37233 requested a full tablet report, sending...
I20260924 02:59:56.619757 14097 ts_manager.cc:194] Registered new tserver with Master: 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559)
I20260924 02:59:56.620105 14097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:49153
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 02:59:56.639123 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37233
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:56.716380 14427 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:56.716568 14427 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:56.716589 14427 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:56.717667 14427 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:56.717700 14427 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:56.717723 14427 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:59:56.719086 14427 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14427
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:56.719272 14427 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:56.719436 14427 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:56.724107 14432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:56.724119 14433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:56.724107 14435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:56.724339 14427 server_base.cc:1034] running on GCE node
I20260924 02:59:56.724567 14427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:56.724784 14427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:56.725941 14427 hybrid_clock.cc:648] HybridClock initialized: now 1790218796725922 us; error 28 us; skew 500 ppm
Sep 24 02:59:56 dist-test-slave-sb6p krb5kdc[14052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:56.728343 14427 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 02:59:56.728822 14427 webserver.cc:460] Webserver started at http://127.0.98.3:33421/ using document root <none> and password file <none>
I20260924 02:59:56.729058 14427 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:56.729112 14427 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:56.729246 14427 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 02:59:56.730233 14427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "d8c7d3c340174c888169ff5545d1787c"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "fd2003ce1a11cc632108e1af0c634a45"
server_key_iv: "2f8fb8b5bdb5e78bc413d37ddb4337b7"
server_key_version: "encryptionkey@0"
I20260924 02:59:56.730597 14427 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "d8c7d3c340174c888169ff5545d1787c"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "fd2003ce1a11cc632108e1af0c634a45"
server_key_iv: "2f8fb8b5bdb5e78bc413d37ddb4337b7"
server_key_version: "encryptionkey@0"
I20260924 02:59:56.731812 14427 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 02:59:56.732529 14442 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:56.732715 14427 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 02:59:56.732795 14427 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "d8c7d3c340174c888169ff5545d1787c"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "fd2003ce1a11cc632108e1af0c634a45"
server_key_iv: "2f8fb8b5bdb5e78bc413d37ddb4337b7"
server_key_version: "encryptionkey@0"
I20260924 02:59:56.732864 14427 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:56.746438 14427 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:56.748227 14427 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:56.748351 14427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:56.748561 14427 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:56.748836 14427 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:59:56.748868 14427 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:56.748896 14427 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:59:56.748919 14427 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:56.754225 14427 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38907
I20260924 02:59:56.754276 14555 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38907 every 8 connection(s)
I20260924 02:59:56.754582 14427 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 02:59:56 dist-test-slave-sb6p krb5kdc[14052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:56.762637 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.755195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43653 (local address 127.0.98.62:37233)
0924 02:59:56.755269 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:56.755271 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:56.755425 (+ 154us) server_negotiation.cc:407] Connection header received
0924 02:59:56.755880 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:56.755881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:56.755891 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:56.755928 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:56.756793 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.757276 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.758357 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.758565 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.758613 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.760508 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:56.760509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:56.760509 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:56.760514 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:56.761553 (+ 1039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:56.761875 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:56.761876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:56.761876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:56.761916 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:56.762104 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:56.762105 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:56.762105 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:56.762204 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:56.762266 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:56.762456 (+ 190us) server_negotiation.cc:299] Negotiation successful
0924 02:59:56.762505 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 02:59:56.762753 14558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.755162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:43653)
0924 02:59:56.755320 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:56.755353 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:56.755798 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:56.755984 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:56.755987 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:56.756545 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:56.756728 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:56.756733 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.757437 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.757439 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:56.758291 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:56.758294 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.758636 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.758637 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:56.758741 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.759176 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:56.759184 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:56.760435 (+ 1251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:56.761620 (+ 1185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:56.761626 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:56.761628 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:56.761822 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:56.761957 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:56.761957 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:56.761957 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:56.762063 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:56.762261 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:56.762263 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:56.762367 (+ 104us) client_negotiation.cc:769] Sending connection context
0924 02:59:56.762440 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 02:59:56.762584 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":81,"threads_started":1}
I20260924 02:59:56.763175 14556 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37233
I20260924 02:59:56.763266 14556 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:56.763309 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 14427
I20260924 02:59:56.763410 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 02:59:56.763443 14556 heartbeater.cc:511] Master 127.0.98.62:37233 requested a full tablet report, sending...
I20260924 02:59:56.763648 392 external_mini_cluster.cc:1428] Setting key d70a29e4303be6490b22cb852649606f
I20260924 02:59:56.763779 14097 ts_manager.cc:194] Registered new tserver with Master: d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907)
I20260924 02:59:56.764236 14097 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43653
I20260924 02:59:56.764452 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 02:59:56.770872 14136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.765593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53992 (local address 127.0.98.62:37233)
0924 02:59:56.765649 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:56.765651 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:56.765717 (+ 66us) server_negotiation.cc:407] Connection header received
0924 02:59:56.765760 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:56.765760 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:56.765767 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:56.765806 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:56.766270 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.766767 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.767508 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.767678 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.767717 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.768777 (+ 1060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:56.768778 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:56.768778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:56.768782 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:56.769825 (+ 1043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:56.770103 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:56.770104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:56.770104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:56.770141 (+ 37us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:56.770335 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:56.770336 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:56.770336 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:56.770453 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:56.770553 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:56.770722 (+ 169us) server_negotiation.cc:299] Negotiation successful
0924 02:59:56.770789 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 02:59:56.772321 14097 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53992:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 02:59:56.773834 14097 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:59:56.779728 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.776129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57182 (local address 127.0.98.3:38907)
0924 02:59:56.776283 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:56.776286 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:56.776293 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:59:56.776324 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:56.776327 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:56.776341 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:56.776440 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:56.776564 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.777632 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.778892 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.779359 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.779421 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.779479 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:56.779609 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 02:59:56.779647 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":96,"threads_started":1}
I20260924 02:59:56.779775 14566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.775755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39559 (local address 127.0.0.1:37370)
0924 02:59:56.776127 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:56.776134 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:56.776171 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:56.776348 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:56.776351 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:56.776353 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:56.776418 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:56.776419 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.777741 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.777742 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:56.778607 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:56.778609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.779483 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.779484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:56.779554 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.779558 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:59:56.779586 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 02:59:56.779659 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":358,"mutex_wait_us":8,"thread_start_us":80,"threads_started":1}
I20260924 02:59:56.779891 14564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.775751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37619 (local address 127.0.0.1:51996)
0924 02:59:56.775910 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:56.775947 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:56.776021 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:56.776613 (+ 592us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:56.776614 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:56.776616 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:56.776654 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:56.776655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.777523 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.777524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:56.778787 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:56.778789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.779695 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.779696 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:56.779760 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.779763 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:59:56.779806 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 02:59:56.779840 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":74,"threads_started":1}
I20260924 02:59:56.780068 14568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.775984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37370 (local address 127.0.98.2:39559)
0924 02:59:56.776120 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:56.776125 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:56.776171 (+ 46us) server_negotiation.cc:407] Connection header received
0924 02:59:56.776238 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:56.776241 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:56.776256 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:56.776334 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:56.776498 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.777616 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.778686 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.779387 (+ 701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.779863 (+ 476us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.779926 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:56.779952 (+ 26us) server_negotiation.cc:299] Negotiation successful
0924 02:59:56.779995 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102,"thread_start_us":59,"threads_started":1}
I20260924 02:59:56.780136 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.775851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51996 (local address 127.0.98.1:37619)
0924 02:59:56.776361 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:56.776364 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:56.776372 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:59:56.776406 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:56.776408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:56.776420 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:56.776483 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:56.776728 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.777435 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.778869 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.779557 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.779611 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.779665 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:56.780024 (+ 359us) server_negotiation.cc:299] Negotiation successful
0924 02:59:56.780065 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":81,"threads_started":1}
I20260924 02:59:56.779776 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.775755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38907 (local address 127.0.0.1:57182)
0924 02:59:56.776009 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:56.776018 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:56.776058 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:56.776446 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:56.776447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:56.776449 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:56.776502 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:56.776504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.777717 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.777717 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:56.778814 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:56.778816 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.779433 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.779434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:56.779512 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.779526 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 02:59:56.779589 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 02:59:56.779688 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"mutex_wait_us":48,"thread_start_us":49,"threads_started":1}
I20260924 02:59:56.780787 14490 tablet_service.cc:1467] Processing CreateTablet for tablet cf75746fae1b47fbbe602cca59e9d892 (DEFAULT_TABLE table=test-table [id=c7d4140037114edc9a26fe4f17f4277a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:59:56.780791 14222 tablet_service.cc:1467] Processing CreateTablet for tablet cf75746fae1b47fbbe602cca59e9d892 (DEFAULT_TABLE table=test-table [id=c7d4140037114edc9a26fe4f17f4277a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:59:56.780781 14356 tablet_service.cc:1467] Processing CreateTablet for tablet cf75746fae1b47fbbe602cca59e9d892 (DEFAULT_TABLE table=test-table [id=c7d4140037114edc9a26fe4f17f4277a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:59:56.781085 14490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf75746fae1b47fbbe602cca59e9d892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:56.781090 14356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf75746fae1b47fbbe602cca59e9d892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:56.781090 14222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cf75746fae1b47fbbe602cca59e9d892. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:56.783703 14570 tablet_bootstrap.cc:492] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c: Bootstrap starting.
I20260924 02:59:56.784164 14571 tablet_bootstrap.cc:492] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47: Bootstrap starting.
I20260924 02:59:56.784590 14570 tablet_bootstrap.cc:654] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:56.784757 14571 tablet_bootstrap.cc:654] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:56.785154 14570 log.cc:824] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:56.785254 14571 log.cc:824] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:56.785545 14572 tablet_bootstrap.cc:492] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147: Bootstrap starting.
I20260924 02:59:56.785861 14570 tablet_bootstrap.cc:492] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c: No bootstrap required, opened a new log
I20260924 02:59:56.785920 14570 ts_tablet_manager.cc:1397] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 02:59:56.786079 14572 tablet_bootstrap.cc:654] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:56.786481 14572 log.cc:824] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:56.787029 14571 tablet_bootstrap.cc:492] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47: No bootstrap required, opened a new log
I20260924 02:59:56.787096 14571 ts_tablet_manager.cc:1397] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260924 02:59:56.787170 14572 tablet_bootstrap.cc:492] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147: No bootstrap required, opened a new log
I20260924 02:59:56.787261 14572 ts_tablet_manager.cc:1397] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260924 02:59:56.787563 14570 raft_consensus.cc:348] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } }
I20260924 02:59:56.787778 14570 raft_consensus.cc:374] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:56.787829 14570 raft_consensus.cc:729] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c7d3c340174c888169ff5545d1787c, State: Initialized, Role: FOLLOWER
I20260924 02:59:56.787946 14570 consensus_queue.cc:260] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } }
I20260924 02:59:56.788209 14570 ts_tablet_manager.cc:1428] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 02:59:56.788218 14556 heartbeater.cc:503] Master 127.0.98.62:37233 was elected leader, sending a full tablet report...
I20260924 02:59:56.788996 14571 raft_consensus.cc:348] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } }
I20260924 02:59:56.789165 14571 raft_consensus.cc:374] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:56.789225 14571 raft_consensus.cc:729] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa85f532f07f4f89bd994bc9b8849e47, State: Initialized, Role: FOLLOWER
I20260924 02:59:56.789363 14571 consensus_queue.cc:260] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } }
I20260924 02:59:56.789350 14572 raft_consensus.cc:348] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } }
I20260924 02:59:56.789510 14572 raft_consensus.cc:374] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:56.789554 14572 raft_consensus.cc:729] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003c7eeafd1646848d258201048b8147, State: Initialized, Role: FOLLOWER
I20260924 02:59:56.789642 14571 ts_tablet_manager.cc:1428] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 02:59:56.789659 14572 consensus_queue.cc:260] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } }
I20260924 02:59:56.789753 14288 heartbeater.cc:503] Master 127.0.98.62:37233 was elected leader, sending a full tablet report...
I20260924 02:59:56.789912 14572 ts_tablet_manager.cc:1428] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260924 02:59:56.790016 14422 heartbeater.cc:503] Master 127.0.98.62:37233 was elected leader, sending a full tablet report...
I20260924 02:59:56.807659 14577 raft_consensus.cc:482] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:59:56.807765 14577 raft_consensus.cc:504] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } }
I20260924 02:59:56.808100 14577 leader_election.cc:288] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559), d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907)
I20260924 02:59:56.810582 14568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.808301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49657 (local address 127.0.98.2:39559)
0924 02:59:56.808370 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:56.808372 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:56.808380 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:59:56.808403 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:56.808405 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:56.808415 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:56.808457 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:56.808583 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.809047 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.809904 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.810262 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.810322 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.810332 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:56.810462 (+ 130us) server_negotiation.cc:299] Negotiation successful
0924 02:59:56.810496 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 02:59:56.810582 14290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.808234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39559 (local address 127.0.98.1:49657)
0924 02:59:56.808284 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:56.808296 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:56.808333 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:56.808467 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:56.808468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:56.808469 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:56.808530 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:56.808532 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.809110 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.809111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:56.809827 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:56.809829 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.810336 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.810337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:56.810405 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.810408 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 02:59:56.810451 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 02:59:56.810489 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 02:59:56.810858 14569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.808301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:48035 (local address 127.0.98.3:38907)
0924 02:59:56.808348 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:56.808350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:56.808447 (+ 97us) server_negotiation.cc:407] Connection header received
0924 02:59:56.808510 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:56.808511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:56.808523 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:56.808568 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:56.808697 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.809169 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.810106 (+ 937us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.810484 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.810589 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.810600 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:56.810752 (+ 152us) server_negotiation.cc:299] Negotiation successful
0924 02:59:56.810793 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 02:59:56.810922 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf75746fae1b47fbbe602cca59e9d892" candidate_uuid: "fa85f532f07f4f89bd994bc9b8849e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003c7eeafd1646848d258201048b8147" is_pre_election: true
I20260924 02:59:56.811055 14579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.808234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38907 (local address 127.0.98.1:48035)
0924 02:59:56.808380 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:56.808390 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:56.808453 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:56.808575 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:56.808576 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:56.808577 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:56.808644 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:56.808646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.809241 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.809241 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:56.810021 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:56.810023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.810592 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.810593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:56.810666 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.810670 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:59:56.810946 (+ 276us) client_negotiation.cc:240] Negotiation successful
0924 02:59:56.810988 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":10,"thread_start_us":78,"threads_started":1}
I20260924 02:59:56.811080 14376 raft_consensus.cc:2454] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa85f532f07f4f89bd994bc9b8849e47 in term 0.
I20260924 02:59:56.811354 14178 leader_election.cc:302] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003c7eeafd1646848d258201048b8147, fa85f532f07f4f89bd994bc9b8849e47; no voters:
I20260924 02:59:56.811388 14510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf75746fae1b47fbbe602cca59e9d892" candidate_uuid: "fa85f532f07f4f89bd994bc9b8849e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8c7d3c340174c888169ff5545d1787c" is_pre_election: true
I20260924 02:59:56.811506 14510 raft_consensus.cc:2454] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa85f532f07f4f89bd994bc9b8849e47 in term 0.
I20260924 02:59:56.811492 14577 raft_consensus.cc:2781] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:59:56.811546 14577 raft_consensus.cc:482] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:59:56.811576 14577 raft_consensus.cc:3037] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:56.812251 14577 raft_consensus.cc:504] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } }
I20260924 02:59:56.812374 14577 leader_election.cc:288] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 1 election: Requested vote from peers 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559), d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907)
I20260924 02:59:56.812521 14510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf75746fae1b47fbbe602cca59e9d892" candidate_uuid: "fa85f532f07f4f89bd994bc9b8849e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8c7d3c340174c888169ff5545d1787c"
I20260924 02:59:56.812569 14376 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cf75746fae1b47fbbe602cca59e9d892" candidate_uuid: "fa85f532f07f4f89bd994bc9b8849e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003c7eeafd1646848d258201048b8147"
I20260924 02:59:56.812592 14510 raft_consensus.cc:3037] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:56.812654 14376 raft_consensus.cc:3037] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:56.813268 14510 raft_consensus.cc:2454] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa85f532f07f4f89bd994bc9b8849e47 in term 1.
I20260924 02:59:56.813309 14376 raft_consensus.cc:2454] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa85f532f07f4f89bd994bc9b8849e47 in term 1.
I20260924 02:59:56.813421 14178 leader_election.cc:302] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003c7eeafd1646848d258201048b8147, fa85f532f07f4f89bd994bc9b8849e47; no voters:
I20260924 02:59:56.813524 14577 raft_consensus.cc:2781] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:59:56.813661 14577 raft_consensus.cc:686] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [term 1 LEADER]: Becoming Leader. State: Replica: fa85f532f07f4f89bd994bc9b8849e47, State: Running, Role: LEADER
I20260924 02:59:56.813755 14577 consensus_queue.cc:237] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } }
I20260924 02:59:56.814589 14096 catalog_manager.cc:5495] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa85f532f07f4f89bd994bc9b8849e47 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fa85f532f07f4f89bd994bc9b8849e47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } health_report { overall_health: UNKNOWN } } }
I20260924 02:59:56.825867 14567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.823814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52006 (local address 127.0.98.1:37619)
0924 02:59:56.823853 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:56.823855 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:56.823863 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:59:56.823920 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:56.823920 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:56.823936 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:56.823987 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:59:56.824068 (+ 81us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.824501 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.825219 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:56.825406 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:56.825453 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:56.825611 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:59:56.825708 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:59:56.825786 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:56.825798 (+ 12us) server_negotiation.cc:299] Negotiation successful
0924 02:59:56.825815 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 02:59:56.827431 14510 raft_consensus.cc:1261] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c [term 1 FOLLOWER]: Refusing update from remote peer fa85f532f07f4f89bd994bc9b8849e47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:59:56.827737 14376 raft_consensus.cc:1261] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147 [term 1 FOLLOWER]: Refusing update from remote peer fa85f532f07f4f89bd994bc9b8849e47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:59:56.827761 14580 consensus_queue.cc:1035] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:59:56.827976 14580 consensus_queue.cc:1035] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:59:56.829869 14583 mvcc.cc:205] Tried to move back new op lower bound from 7332736191802871808 to 7332736191749591040. Current Snapshot: MvccSnapshot[applied={T|T < 7332736191802871808}]
I20260924 02:59:56.830085 14584 mvcc.cc:205] Tried to move back new op lower bound from 7332736191802871808 to 7332736191749591040. Current Snapshot: MvccSnapshot[applied={T|T < 7332736191802871808}]
I20260924 02:59:56.830237 14586 mvcc.cc:205] Tried to move back new op lower bound from 7332736191802871808 to 7332736191749591040. Current Snapshot: MvccSnapshot[applied={T|T < 7332736191802871808}]
I20260924 02:59:56.832407 14096 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:59:56.832504 14096 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53992:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:59:56.833274 14096 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362: Sending DeleteTablet for 3 replicas of tablet cf75746fae1b47fbbe602cca59e9d892
I20260924 02:59:56.833653 14222 tablet_service.cc:1514] Processing DeleteTablet for tablet cf75746fae1b47fbbe602cca59e9d892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:59:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:51996
I20260924 02:59:56.833653 14356 tablet_service.cc:1514] Processing DeleteTablet for tablet cf75746fae1b47fbbe602cca59e9d892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:59:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37370
I20260924 02:59:56.833667 14490 tablet_service.cc:1514] Processing DeleteTablet for tablet cf75746fae1b47fbbe602cca59e9d892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:59:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57182
I20260924 02:59:56.833873 14594 tablet_replica.cc:330] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47: stopping tablet replica
I20260924 02:59:56.833905 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 14159
I20260924 02:59:56.834425 14595 tablet_replica.cc:330] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c: stopping tablet replica
I20260924 02:59:56.834496 14593 tablet_replica.cc:330] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147: stopping tablet replica
I20260924 02:59:56.834544 14595 raft_consensus.cc:2229] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:59:56.834637 14595 raft_consensus.cc:2258] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:59:56.834632 14593 raft_consensus.cc:2229] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:59:56.834733 14593 raft_consensus.cc:2258] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:59:56.835100 14595 ts_tablet_manager.cc:1905] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:59:56.836746 14595 ts_tablet_manager.cc:1918] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:59:56.836815 14595 log.cc:1196] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/cf75746fae1b47fbbe602cca59e9d892
I20260924 02:59:56.837061 14595 ts_tablet_manager.cc:1939] T cf75746fae1b47fbbe602cca59e9d892 P d8c7d3c340174c888169ff5545d1787c: Deleting consensus metadata
I20260924 02:59:56.837531 14081 catalog_manager.cc:4841] TS d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907): tablet cf75746fae1b47fbbe602cca59e9d892 (table test-table [id=c7d4140037114edc9a26fe4f17f4277a]) successfully deleted
W20260924 02:59:56.838949 14083 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37619 (error 108)
I20260924 02:59:56.839119 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 14293
I20260924 02:59:56.843050 14565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.842920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39559 (local address 127.0.0.1:37376)
0924 02:59:56.842962 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:56.843000 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:39559: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 02:59:56.843109 14564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:56.843009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37619 (local address 127.0.0.1:52022)
0924 02:59:56.843055 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:56.843079 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 02:59:56.843122 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 14427
W20260924 02:59:56.843236 14083 catalog_manager.cc:4584] TS fa85f532f07f4f89bd994bc9b8849e47 (127.0.98.1:37619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cf75746fae1b47fbbe602cca59e9d892: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37619: connect: Connection refused (error 111)
I20260924 02:59:56.847326 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 14065
I20260924 02:59:56.851908 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37233
--webserver_interface=127.0.98.62
--webserver_port=38659
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37233
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:56.929183 14597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:56.929401 14597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:56.929433 14597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:56.930636 14597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:59:56.930671 14597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:56.930689 14597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:56.930706 14597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:59:56.930720 14597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:59:56.932174 14597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:56.932413 14597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:56.932619 14597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:56.940397 14603 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:56.940424 14606 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:56.940434 14604 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:56.940693 14597 server_base.cc:1034] running on GCE node
I20260924 02:59:56.940897 14597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:56.941140 14597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:56.942268 14597 hybrid_clock.cc:648] HybridClock initialized: now 1790218796942255 us; error 25 us; skew 500 ppm
Sep 24 02:59:56 dist-test-slave-sb6p krb5kdc[14052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:56.944612 14597 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:59:56.944999 14597 webserver.cc:460] Webserver started at http://127.0.98.62:38659/ using document root <none> and password file <none>
I20260924 02:59:56.945205 14597 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:56.945262 14597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:56.946581 14597 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:59:56.947139 14613 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:56.947284 14597 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:59:56.947366 14597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "4b4cbd6845cc487ea0f61fc513b04362"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "ea037272ac09a2779f9a15a263d777cf"
server_key_iv: "72eb0e889d191d5585121b749c8f3ff1"
server_key_version: "encryptionkey@0"
I20260924 02:59:56.947796 14597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:56.975593 14597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:56.977325 14597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:56.977456 14597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:56.981032 14597 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37233
I20260924 02:59:56.981068 14665 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37233 every 8 connection(s)
I20260924 02:59:56.981421 14597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:59:56.983132 14666 sys_catalog.cc:263] Verifying existing consensus state
I20260924 02:59:56.983760 14666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362: Bootstrap starting.
I20260924 02:59:56.986110 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 14597
I20260924 02:59:56.986353 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37619
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37233
--webserver_port=38223
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 02:59:56.991787 14666 log.cc:824] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:56.993568 14666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 02:59:56.993801 14666 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362: Bootstrap complete.
I20260924 02:59:56.995842 14666 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } }
I20260924 02:59:56.996065 14666 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4cbd6845cc487ea0f61fc513b04362, State: Initialized, Role: FOLLOWER
I20260924 02:59:56.996228 14666 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } }
I20260924 02:59:56.996300 14666 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:59:56.996351 14666 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:59:56.996397 14666 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 1 FOLLOWER]: Advancing to term 2
I20260924 02:59:56.997365 14666 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } }
I20260924 02:59:56.997447 14666 leader_election.cc:302] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b4cbd6845cc487ea0f61fc513b04362; no voters:
I20260924 02:59:56.997591 14666 leader_election.cc:288] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 02:59:56.997649 14670 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 02:59:56.997826 14666 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:59:56.997838 14670 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 2 LEADER]: Becoming Leader. State: Replica: 4b4cbd6845cc487ea0f61fc513b04362, State: Running, Role: LEADER
I20260924 02:59:56.998072 14670 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } }
I20260924 02:59:56.998351 14672 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b4cbd6845cc487ea0f61fc513b04362. Latest consensus state: current_term: 2 leader_uuid: "4b4cbd6845cc487ea0f61fc513b04362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } } }
I20260924 02:59:56.998415 14672 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [sys.catalog]: This master's current role is: LEADER
I20260924 02:59:56.998579 14671 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4b4cbd6845cc487ea0f61fc513b04362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } } }
I20260924 02:59:56.998623 14671 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [sys.catalog]: This master's current role is: LEADER
I20260924 02:59:56.998759 14679 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:59:56.999503 14679 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:59:56.999792 14679 catalog_manager.cc:1259] Loaded cluster ID: cba6ce4fd0a24cebb0419f12450fd020
I20260924 02:59:56.999917 14679 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 02:59:57.000165 14679 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260924 02:59:57.005378 14679 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260924 02:59:57.007740 14679 catalog_manager.cc:1504] Loading token signing keys...
I20260924 02:59:57.007939 14679 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362: Loaded TSK: 0
I20260924 02:59:57.008142 14679 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 02:59:57.068291 14668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:57.068487 14668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:57.068508 14668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:57.069625 14668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:57.069672 14668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:57.069697 14668 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:59:57.071251 14668 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38223
--tserver_master_addrs=127.0.98.62:37233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:57.071499 14668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:57.071744 14668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:57.076541 14693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:57.076503 14694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:57.076704 14696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:57.076606 14668 server_base.cc:1034] running on GCE node
I20260924 02:59:57.076943 14668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:57.077162 14668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:57.078305 14668 hybrid_clock.cc:648] HybridClock initialized: now 1790218797078290 us; error 31 us; skew 500 ppm
Sep 24 02:59:57 dist-test-slave-sb6p krb5kdc[14052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:57.080864 14668 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 02:59:57.081382 14668 webserver.cc:460] Webserver started at http://127.0.98.1:38223/ using document root <none> and password file <none>
I20260924 02:59:57.081640 14668 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:57.081730 14668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:57.083400 14668 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260924 02:59:57.084110 14703 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:57.084281 14668 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:57.084365 14668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "fa85f532f07f4f89bd994bc9b8849e47"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "69d51b78ab3bcfecbde7066c2c8f1bb4"
server_key_iv: "5bfe85a8cfe30143fff7a5f8f2af49ff"
server_key_version: "encryptionkey@0"
I20260924 02:59:57.084692 14668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:57.114593 14668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:57.116648 14668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:57.116784 14668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:57.116992 14668 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:57.117473 14710 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 02:59:57.119247 14668 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 02:59:57.119288 14668 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 02:59:57.119330 14668 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 02:59:57.119917 14668 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 02:59:57.119947 14668 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 02:59:57.120028 14710 tablet_bootstrap.cc:492] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47: Bootstrap starting.
I20260924 02:59:57.125569 14668 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37619
I20260924 02:59:57.125648 14817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37619 every 8 connection(s)
I20260924 02:59:57.125934 14668 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:59:57.129962 14710 log.cc:824] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:57.131409 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 14668
I20260924 02:59:57.131650 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39559
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37233
--webserver_port=39783
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 02:59:57.132203 14710 tablet_bootstrap.cc:492] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 02:59:57.132493 14710 tablet_bootstrap.cc:492] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47: Bootstrap complete.
I20260924 02:59:57.133531 14710 ts_tablet_manager.cc:1397] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.008s
Sep 24 02:59:57 dist-test-slave-sb6p krb5kdc[14052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:57.134644 14710 raft_consensus.cc:348] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } }
I20260924 02:59:57.134959 14710 raft_consensus.cc:729] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa85f532f07f4f89bd994bc9b8849e47, State: Initialized, Role: FOLLOWER
I20260924 02:59:57.135121 14710 consensus_queue.cc:260] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } }
I20260924 02:59:57.135383 14710 ts_tablet_manager.cc:1428] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260924 02:59:57.137946 14820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.126570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:60377)
0924 02:59:57.126759 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:57.126819 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:57.127500 (+ 681us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:57.128226 (+ 726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:57.128230 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:57.128469 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:57.128647 (+ 178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:57.128651 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.130275 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.130276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:57.131146 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:57.131149 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.131986 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.131988 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:57.132136 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.132695 (+ 559us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:57.132706 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:57.134116 (+ 1410us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:57.136276 (+ 2160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:57.136283 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:57.136285 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:57.136653 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:57.136877 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:57.136878 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:57.136878 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:57.137030 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:57.137497 (+ 467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:57.137502 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:57.137613 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 02:59:57.137696 (+ 83us) client_negotiation.cc:240] Negotiation successful
0924 02:59:57.137815 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260924 02:59:57.138106 14821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.126680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60377 (local address 127.0.98.62:37233)
0924 02:59:57.127748 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:57.127758 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:57.127802 (+ 44us) server_negotiation.cc:407] Connection header received
0924 02:59:57.127970 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:57.127977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:57.128040 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:57.128205 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:57.128756 (+ 551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.130177 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.131252 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.131834 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.131971 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.134221 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:57.134225 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:57.134225 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:57.134238 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:57.136179 (+ 1941us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:57.136732 (+ 553us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:57.136734 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:57.136735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:57.136819 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:57.137111 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:57.137113 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:57.137113 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:57.137428 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:57.137516 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:57.137773 (+ 257us) server_negotiation.cc:299] Negotiation successful
0924 02:59:57.137918 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1016,"thread_start_us":69,"threads_started":1}
I20260924 02:59:57.138715 14818 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37233
I20260924 02:59:57.138797 14818 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:57.138948 14818 heartbeater.cc:511] Master 127.0.98.62:37233 requested a full tablet report, sending...
I20260924 02:59:57.139616 14630 ts_manager.cc:194] Registered new tserver with Master: fa85f532f07f4f89bd994bc9b8849e47 (127.0.98.1:37619)
I20260924 02:59:57.140785 14630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60377
I20260924 02:59:57.141618 14818 heartbeater.cc:503] Master 127.0.98.62:37233 was elected leader, sending a full tablet report...
Sep 24 02:59:57 dist-test-slave-sb6p krb5kdc[14052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 02:59:57.147171 14826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.140364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52026 (local address 127.0.98.1:37619)
0924 02:59:57.140497 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:57.140502 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:57.140509 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:59:57.140536 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:57.140538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:57.140549 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:57.140605 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:57.140965 (+ 360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.141923 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.142751 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.143032 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.143087 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.144754 (+ 1667us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:57.144761 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:57.144761 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:57.144767 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:57.146089 (+ 1322us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:57.146456 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:57.146459 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:57.146459 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:57.146530 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:57.146721 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:57.146722 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:57.146722 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:57.146839 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:57.146891 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:57.147048 (+ 157us) server_negotiation.cc:299] Negotiation successful
0924 02:59:57.147111 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":65,"threads_started":1}
I20260924 02:59:57.147238 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.140184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37619 (local address 127.0.0.1:52026)
0924 02:59:57.140312 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:57.140342 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:57.140401 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:57.140615 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:57.140619 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:57.140851 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:57.140894 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:57.140896 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.142072 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.142073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:57.142684 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:57.142687 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.143090 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.143093 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:57.143164 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.143546 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:57.143550 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:57.144682 (+ 1132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:57.146160 (+ 1478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:57.146165 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:57.146174 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:57.146390 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:57.146586 (+ 196us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:57.146586 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:57.146587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:57.146658 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:57.146889 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:57.146895 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:57.146943 (+ 48us) client_negotiation.cc:769] Sending connection context
0924 02:59:57.147025 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 02:59:57.147133 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":75,"threads_started":1}
I20260924 02:59:57.147585 14752 tablet_service.cc:1514] Processing DeleteTablet for tablet cf75746fae1b47fbbe602cca59e9d892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:59:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52026
I20260924 02:59:57.148010 14827 tablet_replica.cc:330] stopping tablet replica
I20260924 02:59:57.148093 14827 raft_consensus.cc:2229] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:59:57.148154 14827 raft_consensus.cc:2258] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:59:57.148402 14827 ts_tablet_manager.cc:1905] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:59:57.150342 14827 ts_tablet_manager.cc:1918] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:59:57.150396 14827 log.cc:1196] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal/wals/cf75746fae1b47fbbe602cca59e9d892
I20260924 02:59:57.150702 14827 ts_tablet_manager.cc:1939] T cf75746fae1b47fbbe602cca59e9d892 P fa85f532f07f4f89bd994bc9b8849e47: Deleting consensus metadata
I20260924 02:59:57.151098 14616 catalog_manager.cc:4841] TS fa85f532f07f4f89bd994bc9b8849e47 (127.0.98.1:37619): tablet cf75746fae1b47fbbe602cca59e9d892 (table test-table [id=c7d4140037114edc9a26fe4f17f4277a]) successfully deleted
W20260924 02:59:57.213377 14823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:57.213577 14823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:57.213606 14823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:57.214751 14823 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:57.214789 14823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:57.214824 14823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:59:57.216219 14823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39783
--tserver_master_addrs=127.0.98.62:37233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14823
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:57.216446 14823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:57.216650 14823 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:57.221478 14833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:57.221516 14823 server_base.cc:1034] running on GCE node
W20260924 02:59:57.221660 14832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:57.221467 14835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:57.221869 14823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:57.222110 14823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:57.223244 14823 hybrid_clock.cc:648] HybridClock initialized: now 1790218797223233 us; error 29 us; skew 500 ppm
Sep 24 02:59:57 dist-test-slave-sb6p krb5kdc[14052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:57.225507 14823 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 02:59:57.225893 14823 webserver.cc:460] Webserver started at http://127.0.98.2:39783/ using document root <none> and password file <none>
I20260924 02:59:57.226091 14823 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:57.226145 14823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:57.227434 14823 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:59:57.228025 14842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:57.228190 14823 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:59:57.228262 14823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "003c7eeafd1646848d258201048b8147"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "285d2b73fa1958fb253067b0850c305f"
server_key_iv: "01cbd91d4321f72dce97091b8cee5577"
server_key_version: "encryptionkey@0"
I20260924 02:59:57.228590 14823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:57.257956 14823 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:57.259816 14823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:57.259953 14823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:57.260178 14823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:57.260562 14849 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 02:59:57.262030 14823 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 02:59:57.262069 14823 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 02:59:57.262100 14823 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 02:59:57.262739 14823 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 02:59:57.262770 14823 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.001s
I20260924 02:59:57.262830 14849 tablet_bootstrap.cc:492] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147: Bootstrap starting.
I20260924 02:59:57.269268 14823 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39559
I20260924 02:59:57.269352 14956 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39559 every 8 connection(s)
I20260924 02:59:57.269645 14823 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 02:59:57.271592 14849 log.cc:824] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:57.273736 14849 tablet_bootstrap.cc:492] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 02:59:57.274015 14849 tablet_bootstrap.cc:492] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147: Bootstrap complete.
I20260924 02:59:57.274878 14849 ts_tablet_manager.cc:1397] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.008s
Sep 24 02:59:57 dist-test-slave-sb6p krb5kdc[14052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:57.275775 14849 raft_consensus.cc:348] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } }
I20260924 02:59:57.276036 14849 raft_consensus.cc:729] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003c7eeafd1646848d258201048b8147, State: Initialized, Role: FOLLOWER
I20260924 02:59:57.276171 14849 consensus_queue.cc:260] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } } peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } }
I20260924 02:59:57.276443 14849 ts_tablet_manager.cc:1428] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 02:59:57.278038 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 14823
I20260924 02:59:57.278319 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38907
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37233
--webserver_port=33421
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 02:59:57.278734 14821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.270350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:43459 (local address 127.0.98.62:37233)
0924 02:59:57.270392 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:57.270394 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:57.270642 (+ 248us) server_negotiation.cc:407] Connection header received
0924 02:59:57.271159 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:57.271160 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:57.271178 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:57.271232 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:57.271893 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.272633 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.273736 (+ 1103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.274049 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.274127 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.276165 (+ 2038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:57.276166 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:57.276166 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:57.276175 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:57.277408 (+ 1233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:57.277766 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:57.277767 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:57.277767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:57.277811 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:57.278061 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:57.278062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:57.278062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:57.278194 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:57.278244 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:57.278596 (+ 352us) server_negotiation.cc:299] Negotiation successful
0924 02:59:57.278671 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 02:59:57.279011 14959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.270319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:43459)
0924 02:59:57.270498 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:57.270559 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:57.271020 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:57.271303 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:57.271308 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:57.271556 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:57.271791 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:57.271798 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.272751 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.272751 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:57.273644 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:57.273647 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.274149 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.274150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:57.274249 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.274786 (+ 537us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:57.274794 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:57.276080 (+ 1286us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:57.277479 (+ 1399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:57.277485 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:57.277486 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:57.277715 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:57.277872 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:57.277873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:57.277873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:57.278003 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:57.278239 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:57.278246 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:57.278408 (+ 162us) client_negotiation.cc:769] Sending connection context
0924 02:59:57.278571 (+ 163us) client_negotiation.cc:240] Negotiation successful
0924 02:59:57.278760 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":110,"threads_started":1}
I20260924 02:59:57.279819 14957 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37233
I20260924 02:59:57.279898 14957 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:57.280055 14957 heartbeater.cc:511] Master 127.0.98.62:37233 requested a full tablet report, sending...
I20260924 02:59:57.280529 14630 ts_manager.cc:194] Registered new tserver with Master: 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559)
I20260924 02:59:57.281044 14630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:43459
I20260924 02:59:57.281911 14957 heartbeater.cc:503] Master 127.0.98.62:37233 was elected leader, sending a full tablet report...
Sep 24 02:59:57 dist-test-slave-sb6p krb5kdc[14052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 02:59:57.287389 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.280785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39559 (local address 127.0.0.1:37388)
0924 02:59:57.281149 (+ 364us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:57.281167 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:57.281215 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:57.281344 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:57.281344 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:57.281496 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:57.281539 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:57.281541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.282537 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.282538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:57.283037 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:57.283039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.283457 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.283460 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:57.283532 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.283913 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:57.283917 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:57.284967 (+ 1050us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:57.286429 (+ 1462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:57.286430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:57.286430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:57.286632 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:57.286806 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:57.286807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:57.286807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:57.286862 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:57.287102 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:57.287103 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:57.287144 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 02:59:57.287205 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 02:59:57.287299 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":350}
I20260924 02:59:57.287407 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.280847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.0.98.2:39559)
0924 02:59:57.280989 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:57.280993 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:57.281209 (+ 216us) server_negotiation.cc:407] Connection header received
0924 02:59:57.281272 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:57.281274 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:57.281287 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:57.281345 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:57.281617 (+ 272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.282450 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.283105 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.283397 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.283451 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.285036 (+ 1585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:57.285043 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:57.285043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:57.285050 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:57.286357 (+ 1307us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:57.286698 (+ 341us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:57.286701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:57.286701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:57.286756 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:57.286911 (+ 155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:57.286912 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:57.286912 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:57.287044 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:57.287112 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:57.287244 (+ 132us) server_negotiation.cc:299] Negotiation successful
0924 02:59:57.287325 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":69,"threads_started":1}
I20260924 02:59:57.287740 14891 tablet_service.cc:1514] Processing DeleteTablet for tablet cf75746fae1b47fbbe602cca59e9d892 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:59:56 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37388
I20260924 02:59:57.288182 14964 tablet_replica.cc:330] stopping tablet replica
I20260924 02:59:57.288282 14964 raft_consensus.cc:2229] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:59:57.288370 14964 raft_consensus.cc:2258] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:59:57.288672 14964 ts_tablet_manager.cc:1905] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:59:57.290869 14964 ts_tablet_manager.cc:1918] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:59:57.290938 14964 log.cc:1196] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/cf75746fae1b47fbbe602cca59e9d892
I20260924 02:59:57.291139 14964 ts_tablet_manager.cc:1939] T cf75746fae1b47fbbe602cca59e9d892 P 003c7eeafd1646848d258201048b8147: Deleting consensus metadata
I20260924 02:59:57.291586 14617 catalog_manager.cc:4841] TS 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559): tablet cf75746fae1b47fbbe602cca59e9d892 (table test-table [id=c7d4140037114edc9a26fe4f17f4277a]) successfully deleted
W20260924 02:59:57.358913 14962 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:57.359103 14962 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:57.359123 14962 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:57.360230 14962 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:57.360268 14962 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:57.360296 14962 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:59:57.361658 14962 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33421
--tserver_master_addrs=127.0.98.62:37233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14962
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:57.361864 14962 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:57.362056 14962 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:57.366743 14970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:57.366743 14972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:57.366846 14962 server_base.cc:1034] running on GCE node
W20260924 02:59:57.366830 14969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:57.367154 14962 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:57.367349 14962 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:57.368477 14962 hybrid_clock.cc:648] HybridClock initialized: now 1790218797368464 us; error 25 us; skew 500 ppm
Sep 24 02:59:57 dist-test-slave-sb6p krb5kdc[14052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:57.370725 14962 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 02:59:57.371122 14962 webserver.cc:460] Webserver started at http://127.0.98.3:33421/ using document root <none> and password file <none>
I20260924 02:59:57.371306 14962 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:57.371363 14962 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:57.372548 14962 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 02:59:57.373062 14979 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:57.373207 14962 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:57.373281 14962 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "d8c7d3c340174c888169ff5545d1787c"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "fd2003ce1a11cc632108e1af0c634a45"
server_key_iv: "2f8fb8b5bdb5e78bc413d37ddb4337b7"
server_key_version: "encryptionkey@0"
I20260924 02:59:57.373584 14962 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:57.392764 14962 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:57.394817 14962 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:57.394942 14962 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:57.395143 14962 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:57.395421 14962 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:59:57.395452 14962 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:57.395479 14962 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:59:57.395503 14962 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:57.400854 14962 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38907
I20260924 02:59:57.400923 15092 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38907 every 8 connection(s)
I20260924 02:59:57.401235 14962 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 02:59:57.403013 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 14962
Sep 24 02:59:57 dist-test-slave-sb6p krb5kdc[14052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 02:59:57.409519 15097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.403355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54006 (local address 127.0.98.62:37233)
0924 02:59:57.403444 (+ 89us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:57.403446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:57.403478 (+ 32us) server_negotiation.cc:407] Connection header received
0924 02:59:57.403549 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:57.403550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:57.403558 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:57.403609 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:57.404128 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.404825 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.405732 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.405938 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.405989 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.407206 (+ 1217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:57.407207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:57.407207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:57.407213 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:57.408532 (+ 1319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:57.408860 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:57.408861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:57.408862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:57.408915 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:57.409079 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:57.409079 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:57.409079 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:57.409174 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:57.409291 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:57.409361 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 02:59:57.409425 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71,"thread_start_us":51,"threads_started":1}
I20260924 02:59:57.410638 14821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.401860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43965 (local address 127.0.98.62:37233)
0924 02:59:57.401918 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:57.401920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:57.402117 (+ 197us) server_negotiation.cc:407] Connection header received
0924 02:59:57.402627 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:57.402628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:57.402640 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:57.402689 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:57.403722 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.404256 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.405639 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.405834 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.405889 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.407957 (+ 2068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:57.407959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:57.407959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:57.407965 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:57.409382 (+ 1417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:57.409799 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:57.409800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:57.409800 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:57.409856 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:57.410100 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:57.410101 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:57.410102 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:57.410223 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:57.410279 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:57.410489 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 02:59:57.410576 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 02:59:57.410825 15095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.401853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:43965)
0924 02:59:57.402001 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:57.402042 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:57.402506 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:57.402765 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:57.402768 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:57.403376 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:57.403648 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:57.403655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.404352 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.404354 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:57.405549 (+ 1195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:57.405552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.405900 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.405901 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:57.406025 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.406566 (+ 541us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:57.406576 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:57.407879 (+ 1303us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:57.409451 (+ 1572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:57.409457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:57.409459 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:57.409734 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:57.409913 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:57.409914 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:57.409914 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:57.410041 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:57.410282 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:57.410286 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:57.410398 (+ 112us) client_negotiation.cc:769] Sending connection context
0924 02:59:57.410476 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 02:59:57.410629 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260924 02:59:57.411216 15093 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37233
I20260924 02:59:57.411293 15093 heartbeater.cc:464] Registering TS with master...
I20260924 02:59:57.411463 15093 heartbeater.cc:511] Master 127.0.98.62:37233 requested a full tablet report, sending...
I20260924 02:59:57.411826 14630 ts_manager.cc:194] Registered new tserver with Master: d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907)
I20260924 02:59:57.412259 14630 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:43965
I20260924 02:59:57.420796 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 02:59:57.423303 14821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.421137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54022 (local address 127.0.98.62:37233)
0924 02:59:57.421187 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:57.421189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:57.421304 (+ 115us) server_negotiation.cc:407] Connection header received
0924 02:59:57.421356 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:57.421357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:57.421390 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:57.421445 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:59:57.421524 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.421989 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.422744 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.422919 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.422965 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.423070 (+ 105us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:59:57.423161 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:59:57.423213 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:57.423226 (+ 13us) server_negotiation.cc:299] Negotiation successful
0924 02:59:57.423240 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260924 02:59:57.423568 14630 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:54022:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 02:59:57.425082 14630 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 02:59:57.427013 14752 tablet_service.cc:1467] Processing CreateTablet for tablet 849afbdd4a044004aa4b3ef8907b9acd (DEFAULT_TABLE table=test-table [id=16edb2fcf14242ce8cb0cf122e4fe045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:59:57.427201 14752 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 849afbdd4a044004aa4b3ef8907b9acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:57.428450 14710 tablet_bootstrap.cc:492] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47: Bootstrap starting.
I20260924 02:59:57.428596 14891 tablet_service.cc:1467] Processing CreateTablet for tablet 849afbdd4a044004aa4b3ef8907b9acd (DEFAULT_TABLE table=test-table [id=16edb2fcf14242ce8cb0cf122e4fe045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:59:57.428748 14891 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 849afbdd4a044004aa4b3ef8907b9acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:57.428758 14710 tablet_bootstrap.cc:654] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:57.429332 14710 tablet_bootstrap.cc:492] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47: No bootstrap required, opened a new log
I20260924 02:59:57.429371 14710 ts_tablet_manager.cc:1397] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260924 02:59:57.429728 14710 raft_consensus.cc:348] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } } peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } }
I20260924 02:59:57.429806 14710 raft_consensus.cc:374] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:57.429850 14710 raft_consensus.cc:729] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa85f532f07f4f89bd994bc9b8849e47, State: Initialized, Role: FOLLOWER
I20260924 02:59:57.429898 14710 consensus_queue.cc:260] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } } peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } }
I20260924 02:59:57.429989 14710 ts_tablet_manager.cc:1428] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 02:59:57.430096 14849 tablet_bootstrap.cc:492] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147: Bootstrap starting.
I20260924 02:59:57.430369 14849 tablet_bootstrap.cc:654] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:57.430778 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.426770 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38907 (local address 127.0.0.1:57190)
0924 02:59:57.426899 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:57.426908 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:57.426973 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:57.427432 (+ 459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:57.427433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:57.427434 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:57.427485 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:57.427487 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.428428 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.428429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:57.429539 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:57.429541 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.430501 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.430502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:57.430591 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.430613 (+ 22us) client_negotiation.cc:769] Sending connection context
0924 02:59:57.430670 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 02:59:57.430713 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114}
I20260924 02:59:57.430845 14849 tablet_bootstrap.cc:492] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147: No bootstrap required, opened a new log
I20260924 02:59:57.430886 14849 ts_tablet_manager.cc:1397] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260924 02:59:57.431041 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.427059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57190 (local address 127.0.98.3:38907)
0924 02:59:57.427273 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:57.427278 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:57.427288 (+ 10us) server_negotiation.cc:407] Connection header received
0924 02:59:57.427326 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:57.427328 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:57.427343 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:57.427414 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:57.427536 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.428342 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.429621 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.430347 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.430405 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.430469 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:57.430906 (+ 437us) server_negotiation.cc:299] Negotiation successful
0924 02:59:57.430949 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":177,"thread_start_us":133,"threads_started":1}
I20260924 02:59:57.431183 14849 raft_consensus.cc:348] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } } peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } }
I20260924 02:59:57.431253 14849 raft_consensus.cc:374] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:57.431295 14849 raft_consensus.cc:729] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 003c7eeafd1646848d258201048b8147, State: Initialized, Role: FOLLOWER
I20260924 02:59:57.431344 14849 consensus_queue.cc:260] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } } peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } }
I20260924 02:59:57.431429 14849 ts_tablet_manager.cc:1428] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 02:59:57.431677 15027 tablet_service.cc:1467] Processing CreateTablet for tablet 849afbdd4a044004aa4b3ef8907b9acd (DEFAULT_TABLE table=test-table [id=16edb2fcf14242ce8cb0cf122e4fe045]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 02:59:57.432027 15027 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 849afbdd4a044004aa4b3ef8907b9acd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 02:59:57.434676 15102 tablet_bootstrap.cc:492] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c: Bootstrap starting.
I20260924 02:59:57.435190 15102 tablet_bootstrap.cc:654] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c: Neither blocks nor log segments found. Creating new log.
I20260924 02:59:57.435689 15102 log.cc:824] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:57.436304 15102 tablet_bootstrap.cc:492] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c: No bootstrap required, opened a new log
I20260924 02:59:57.436363 15102 ts_tablet_manager.cc:1397] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 02:59:57.437623 15102 raft_consensus.cc:348] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } } peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } }
I20260924 02:59:57.437762 15102 raft_consensus.cc:374] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 02:59:57.437793 15102 raft_consensus.cc:729] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8c7d3c340174c888169ff5545d1787c, State: Initialized, Role: FOLLOWER
I20260924 02:59:57.437906 15102 consensus_queue.cc:260] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } } peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } }
I20260924 02:59:57.438145 15102 ts_tablet_manager.cc:1428] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 02:59:57.438180 15093 heartbeater.cc:503] Master 127.0.98.62:37233 was elected leader, sending a full tablet report...
W20260924 02:59:57.520200 14958 tablet.cc:2369] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 02:59:57.586403 14824 raft_consensus.cc:482] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:59:57.586556 14824 raft_consensus.cc:504] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } } peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } }
I20260924 02:59:57.586874 14824 leader_election.cc:288] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907), 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559)
I20260924 02:59:57.589617 15099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.587144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41111 (local address 127.0.98.3:38907)
0924 02:59:57.587206 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:57.587208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:57.587216 (+ 8us) server_negotiation.cc:407] Connection header received
0924 02:59:57.587242 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:57.587243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:57.587253 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:57.587305 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:57.587451 (+ 146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.587966 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.588938 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.589312 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.589368 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.589378 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:57.589520 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 02:59:57.589554 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 02:59:57.589630 14963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.587136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57667 (local address 127.0.98.2:39559)
0924 02:59:57.587188 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:57.587190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:57.587224 (+ 34us) server_negotiation.cc:407] Connection header received
0924 02:59:57.587297 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:57.587298 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:57.587317 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:57.587369 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 02:59:57.587521 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.588004 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.588942 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.589327 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.589386 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.589459 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:57.589528 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 02:59:57.589563 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 02:59:57.589639 14820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.587040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38907 (local address 127.0.98.1:41111)
0924 02:59:57.587090 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:57.587103 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:57.587158 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:57.587312 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:57.587313 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:57.587315 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:57.587390 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:57.587392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.588027 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.588027 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:57.588858 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:57.588860 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.589378 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.589379 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:57.589444 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.589459 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 02:59:57.589503 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 02:59:57.589542 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29,"mutex_wait_us":2}
I20260924 02:59:57.589870 15105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.587058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39559 (local address 127.0.98.1:57667)
0924 02:59:57.587175 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:57.587185 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:57.587236 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:57.587376 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:57.587377 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:57.587378 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 02:59:57.587448 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:57.587449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.588071 (+ 622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.588072 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:57.588857 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:57.588859 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.589412 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.589413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:57.589474 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.589478 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 02:59:57.589524 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 02:59:57.589560 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":2,"thread_start_us":57,"threads_started":1}
I20260924 02:59:57.589968 15047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "849afbdd4a044004aa4b3ef8907b9acd" candidate_uuid: "fa85f532f07f4f89bd994bc9b8849e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8c7d3c340174c888169ff5545d1787c" is_pre_election: true
I20260924 02:59:57.590126 15047 raft_consensus.cc:2454] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa85f532f07f4f89bd994bc9b8849e47 in term 0.
I20260924 02:59:57.590135 14911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "849afbdd4a044004aa4b3ef8907b9acd" candidate_uuid: "fa85f532f07f4f89bd994bc9b8849e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003c7eeafd1646848d258201048b8147" is_pre_election: true
I20260924 02:59:57.590243 14911 raft_consensus.cc:2454] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa85f532f07f4f89bd994bc9b8849e47 in term 0.
I20260924 02:59:57.590318 14704 leader_election.cc:302] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d8c7d3c340174c888169ff5545d1787c, fa85f532f07f4f89bd994bc9b8849e47; no voters:
I20260924 02:59:57.590451 14824 raft_consensus.cc:2781] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 02:59:57.590505 14824 raft_consensus.cc:482] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 02:59:57.590557 14824 raft_consensus.cc:3037] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:57.591208 14824 raft_consensus.cc:504] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } } peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } }
I20260924 02:59:57.591318 14824 leader_election.cc:288] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 1 election: Requested vote from peers d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907), 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559)
I20260924 02:59:57.591445 14911 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "849afbdd4a044004aa4b3ef8907b9acd" candidate_uuid: "fa85f532f07f4f89bd994bc9b8849e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "003c7eeafd1646848d258201048b8147"
I20260924 02:59:57.591497 15047 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "849afbdd4a044004aa4b3ef8907b9acd" candidate_uuid: "fa85f532f07f4f89bd994bc9b8849e47" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8c7d3c340174c888169ff5545d1787c"
I20260924 02:59:57.591519 14911 raft_consensus.cc:3037] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147 [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:57.591566 15047 raft_consensus.cc:3037] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c [term 0 FOLLOWER]: Advancing to term 1
I20260924 02:59:57.592185 14911 raft_consensus.cc:2454] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa85f532f07f4f89bd994bc9b8849e47 in term 1.
I20260924 02:59:57.592197 15047 raft_consensus.cc:2454] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa85f532f07f4f89bd994bc9b8849e47 in term 1.
I20260924 02:59:57.592344 14707 leader_election.cc:302] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 003c7eeafd1646848d258201048b8147, fa85f532f07f4f89bd994bc9b8849e47; no voters:
I20260924 02:59:57.592432 14824 raft_consensus.cc:2781] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 02:59:57.592572 14824 raft_consensus.cc:686] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 LEADER]: Becoming Leader. State: Replica: fa85f532f07f4f89bd994bc9b8849e47, State: Running, Role: LEADER
I20260924 02:59:57.592628 14824 consensus_queue.cc:237] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } } peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } }
I20260924 02:59:57.593338 14630 catalog_manager.cc:5495] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 reported cstate change: term changed from 0 to 1, leader changed from <none> to fa85f532f07f4f89bd994bc9b8849e47 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "fa85f532f07f4f89bd994bc9b8849e47" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } health_report { overall_health: UNKNOWN } } }
I20260924 02:59:57.615265 14826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.613068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52042 (local address 127.0.98.1:37619)
0924 02:59:57.613111 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:57.613113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:57.613120 (+ 7us) server_negotiation.cc:407] Connection header received
0924 02:59:57.613186 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:57.613187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:57.613203 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:57.613258 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 02:59:57.613367 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.613848 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.614507 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:57.614769 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:57.614800 (+ 31us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:57.615011 (+ 211us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 02:59:57.615105 (+ 94us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 02:59:57.615155 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:57.615192 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 02:59:57.615209 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 02:59:57.616647 14911 raft_consensus.cc:1261] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147 [term 1 FOLLOWER]: Refusing update from remote peer fa85f532f07f4f89bd994bc9b8849e47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:59:57.616644 15047 raft_consensus.cc:1261] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c [term 1 FOLLOWER]: Refusing update from remote peer fa85f532f07f4f89bd994bc9b8849e47: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 02:59:57.616967 15106 consensus_queue.cc:1035] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:59:57.617094 14824 consensus_queue.cc:1035] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 02:59:57.619740 15110 mvcc.cc:205] Tried to move back new op lower bound from 7332736195035717632 to 7332736194939838464. Current Snapshot: MvccSnapshot[applied={T|T < 7332736195035717632}]
I20260924 02:59:57.621497 14629 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:54022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:59:57.621579 14629 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:54022:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 02:59:57.622253 14629 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362: Sending DeleteTablet for 3 replicas of tablet 849afbdd4a044004aa4b3ef8907b9acd
I20260924 02:59:57.622563 14752 tablet_service.cc:1514] Processing DeleteTablet for tablet 849afbdd4a044004aa4b3ef8907b9acd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:59:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:52026
I20260924 02:59:57.622624 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 14668
I20260924 02:59:57.622611 15027 tablet_service.cc:1514] Processing DeleteTablet for tablet 849afbdd4a044004aa4b3ef8907b9acd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:59:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:57190
I20260924 02:59:57.622658 14827 tablet_replica.cc:330] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47: stopping tablet replica
I20260924 02:59:57.622699 14827 raft_consensus.cc:2229] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 LEADER]: Raft consensus shutting down.
I20260924 02:59:57.622771 14827 raft_consensus.cc:2258] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:59:57.624377 14891 tablet_service.cc:1514] Processing DeleteTablet for tablet 849afbdd4a044004aa4b3ef8907b9acd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 02:59:57 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:37388
I20260924 02:59:57.624488 14964 tablet_replica.cc:330] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147: stopping tablet replica
I20260924 02:59:57.624552 14964 raft_consensus.cc:2229] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:59:57.624604 14964 raft_consensus.cc:2258] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:59:57.624862 14964 ts_tablet_manager.cc:1905] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:59:57.626420 14964 ts_tablet_manager.cc:1918] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:59:57.626485 14964 log.cc:1196] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/849afbdd4a044004aa4b3ef8907b9acd
I20260924 02:59:57.626696 14964 ts_tablet_manager.cc:1939] T 849afbdd4a044004aa4b3ef8907b9acd P 003c7eeafd1646848d258201048b8147: Deleting consensus metadata
I20260924 02:59:57.626958 14617 catalog_manager.cc:4841] TS 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559): tablet 849afbdd4a044004aa4b3ef8907b9acd (table test-table [id=16edb2fcf14242ce8cb0cf122e4fe045]) successfully deleted
W20260924 02:59:57.627863 14616 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:37619 (error 108)
I20260924 02:59:57.628093 15118 tablet_replica.cc:330] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c: stopping tablet replica
I20260924 02:59:57.628094 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 14823
I20260924 02:59:57.628600 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.628193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:37619 (local address 127.0.0.1:52058)
0924 02:59:57.628538 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:57.628570 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37619: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327}
I20260924 02:59:57.628623 15118 raft_consensus.cc:2229] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 02:59:57.628729 15118 raft_consensus.cc:2258] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 02:59:57.629144 15118 ts_tablet_manager.cc:1905] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 02:59:57.630600 15118 ts_tablet_manager.cc:1918] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 02:59:57.630671 15118 log.cc:1196] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/849afbdd4a044004aa4b3ef8907b9acd
I20260924 02:59:57.630925 15118 ts_tablet_manager.cc:1939] T 849afbdd4a044004aa4b3ef8907b9acd P d8c7d3c340174c888169ff5545d1787c: Deleting consensus metadata
W20260924 02:59:57.630699 14616 catalog_manager.cc:4584] TS fa85f532f07f4f89bd994bc9b8849e47 (127.0.98.1:37619): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 849afbdd4a044004aa4b3ef8907b9acd: Network error: Client connection negotiation failed: client connection to 127.0.98.1:37619: connect: Connection refused (error 111)
I20260924 02:59:57.631313 14614 catalog_manager.cc:4841] TS d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907): tablet 849afbdd4a044004aa4b3ef8907b9acd (table test-table [id=16edb2fcf14242ce8cb0cf122e4fe045]) successfully deleted
I20260924 02:59:57.633767 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 14962
I20260924 02:59:57.637863 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 14597
I20260924 02:59:57.642113 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37233
--webserver_interface=127.0.98.62
--webserver_port=38659
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37233
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:57.720342 15120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:57.720547 15120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:57.720569 15120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:57.721722 15120 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 02:59:57.721762 15120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:57.721776 15120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:57.721798 15120 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 02:59:57.721827 15120 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 02:59:57.723594 15120 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37233
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37233
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=38659
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:57.723852 15120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:57.724086 15120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 02:59:57.728807 15120 server_base.cc:1034] running on GCE node
W20260924 02:59:57.728780 15126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:57.728806 15125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:57.728776 15128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:57.729224 15120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:57.729480 15120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:57.730587 15120 hybrid_clock.cc:648] HybridClock initialized: now 1790218797730570 us; error 25 us; skew 500 ppm
Sep 24 02:59:57 dist-test-slave-sb6p krb5kdc[14052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:57.732848 15120 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 02:59:57.733263 15120 webserver.cc:460] Webserver started at http://127.0.98.62:38659/ using document root <none> and password file <none>
I20260924 02:59:57.733464 15120 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:57.733511 15120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:57.734786 15120 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:59:57.735327 15135 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:57.735460 15120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:59:57.735538 15120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "4b4cbd6845cc487ea0f61fc513b04362"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "ea037272ac09a2779f9a15a263d777cf"
server_key_iv: "72eb0e889d191d5585121b749c8f3ff1"
server_key_version: "encryptionkey@0"
I20260924 02:59:57.735853 15120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:57.752162 15120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:57.753829 15120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:57.753957 15120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:57.757469 15120 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37233
I20260924 02:59:57.757506 15187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37233 every 8 connection(s)
I20260924 02:59:57.757828 15120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 02:59:57.759294 15188 sys_catalog.cc:263] Verifying existing consensus state
I20260924 02:59:57.759907 15188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362: Bootstrap starting.
I20260924 02:59:57.766428 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 15120
I20260924 02:59:57.766752 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:37619
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37233
--webserver_port=38223
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 02:59:57.767966 15188 log.cc:824] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:57.770118 15188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 02:59:57.770368 15188 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362: Bootstrap complete.
I20260924 02:59:57.772405 15188 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } }
I20260924 02:59:57.772614 15188 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b4cbd6845cc487ea0f61fc513b04362, State: Initialized, Role: FOLLOWER
I20260924 02:59:57.772759 15188 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } }
I20260924 02:59:57.772821 15188 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 02:59:57.772859 15188 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 02:59:57.772904 15188 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 2 FOLLOWER]: Advancing to term 3
I20260924 02:59:57.773828 15188 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } }
I20260924 02:59:57.773911 15188 leader_election.cc:302] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b4cbd6845cc487ea0f61fc513b04362; no voters:
I20260924 02:59:57.774053 15188 leader_election.cc:288] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 02:59:57.774097 15192 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 02:59:57.774224 15192 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [term 3 LEADER]: Becoming Leader. State: Replica: 4b4cbd6845cc487ea0f61fc513b04362, State: Running, Role: LEADER
I20260924 02:59:57.774308 15188 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 02:59:57.774330 15192 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } }
I20260924 02:59:57.774616 15192 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b4cbd6845cc487ea0f61fc513b04362. Latest consensus state: current_term: 3 leader_uuid: "4b4cbd6845cc487ea0f61fc513b04362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } } }
I20260924 02:59:57.774701 15192 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [sys.catalog]: This master's current role is: LEADER
I20260924 02:59:57.774927 15193 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "4b4cbd6845cc487ea0f61fc513b04362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b4cbd6845cc487ea0f61fc513b04362" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37233 } } }
I20260924 02:59:57.775033 15193 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b4cbd6845cc487ea0f61fc513b04362 [sys.catalog]: This master's current role is: LEADER
I20260924 02:59:57.775135 15198 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 02:59:57.775923 15198 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 02:59:57.776016 15198 catalog_manager.cc:1259] Loaded cluster ID: cba6ce4fd0a24cebb0419f12450fd020
I20260924 02:59:57.776044 15198 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260924 02:59:57.776249 15198 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fad065b99ad google::LogMessage::Fail() at ??:0
@ 0x7fad065bc88a google::LogMessage::SendToLog() at ??:0
@ 0x7fad065b94de google::LogMessage::Flush() at ??:0
@ 0x7fad065baa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fad0e4da0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fad0e4e38b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fad084870d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fad0847c954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fad092e06db start_thread at ??:0
@ 0x7fad050b761f clone at ??:0
W20260924 02:59:57.874257 15190 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:57.874446 15190 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:57.874467 15190 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:57.875626 15190 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:57.875682 15190 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:57.875720 15190 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 02:59:57.877151 15190 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:37619
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=38223
--tserver_master_addrs=127.0.98.62:37233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15190
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:57.877395 15190 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:57.877595 15190 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:57.882282 15227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:57.882318 15225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:57.882278 15224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:57.882622 15190 server_base.cc:1034] running on GCE node
I20260924 02:59:57.882843 15190 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:57.883111 15190 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:57.884284 15190 hybrid_clock.cc:648] HybridClock initialized: now 1790218797884271 us; error 27 us; skew 500 ppm
Sep 24 02:59:57 dist-test-slave-sb6p krb5kdc[14052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:57.887120 15190 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 02:59:57.887658 15190 webserver.cc:460] Webserver started at http://127.0.98.1:38223/ using document root <none> and password file <none>
I20260924 02:59:57.887926 15190 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:57.888000 15190 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:57.889377 15190 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:59:57.890040 15234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:57.890197 15190 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 02:59:57.890272 15190 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "fa85f532f07f4f89bd994bc9b8849e47"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "69d51b78ab3bcfecbde7066c2c8f1bb4"
server_key_iv: "5bfe85a8cfe30143fff7a5f8f2af49ff"
server_key_version: "encryptionkey@0"
I20260924 02:59:57.890625 15190 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:57.899101 15190 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:57.900991 15190 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:57.901095 15190 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:57.901301 15190 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:57.901691 15241 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 02:59:57.903075 15190 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 02:59:57.903115 15190 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 02:59:57.903154 15190 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 02:59:57.903760 15190 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 02:59:57.903800 15190 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 02:59:57.903872 15241 tablet_bootstrap.cc:492] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47: Bootstrap starting.
I20260924 02:59:57.911070 15190 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:37619
I20260924 02:59:57.911118 15348 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:37619 every 8 connection(s)
I20260924 02:59:57.911535 15190 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 02:59:57.911840 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 15190
I20260924 02:59:57.912114 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:39559
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37233
--webserver_port=39783
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 02:59:57.914076 15241 log.cc:824] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47: Log is configured to *not* fsync() on all Append() calls
I20260924 02:59:57.917052 15241 tablet_bootstrap.cc:492] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 02:59:57.917348 15241 tablet_bootstrap.cc:492] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47: Bootstrap complete.
I20260924 02:59:57.918406 15241 ts_tablet_manager.cc:1397] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.007s
I20260924 02:59:57.919756 15241 raft_consensus.cc:348] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } } peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } }
I20260924 02:59:57.920063 15241 raft_consensus.cc:729] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa85f532f07f4f89bd994bc9b8849e47, State: Initialized, Role: FOLLOWER
I20260924 02:59:57.920228 15241 consensus_queue.cc:260] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } } peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } }
I20260924 02:59:57.920486 15241 ts_tablet_manager.cc:1428] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260924 02:59:57.985999 15352 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 02:59:57.913782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:57895)
0924 02:59:57.914339 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:57.914386 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:57.915961 (+ 1575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:57.985797 (+ 69836us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":83,"threads_started":1}
W20260924 02:59:57.986268 15349 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37233 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37233: Client connection negotiation failed: client connection to 127.0.98.62:37233: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 02:59:57.986609 15352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.986470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:38233)
0924 02:59:57.986536 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:57.986582 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 02:59:57.986929 15352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:57.986840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:46005)
0924 02:59:57.986883 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:57.986909 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260924 02:59:57.987042 15349 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 02:59:57.991214 15351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:57.991389 15351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:57.991418 15351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:57.992532 15351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:57.992568 15351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:57.992602 15351 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 02:59:57.993992 15351 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:39559
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=39783
--tserver_master_addrs=127.0.98.62:37233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:57.994222 15351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:57.994411 15351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:57.998868 15362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:57.998881 15359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:57.998878 15360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:57.999155 15351 server_base.cc:1034] running on GCE node
I20260924 02:59:57.999316 15351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:57.999503 15351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:58.000628 15351 hybrid_clock.cc:648] HybridClock initialized: now 1790218798000613 us; error 28 us; skew 500 ppm
Sep 24 02:59:57 dist-test-slave-sb6p krb5kdc[14052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:58.002858 15351 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 02:59:58.003252 15351 webserver.cc:460] Webserver started at http://127.0.98.2:39783/ using document root <none> and password file <none>
I20260924 02:59:58.003450 15351 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:58.003491 15351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:58.004668 15351 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 02:59:58.005115 15369 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:58.005250 15351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:58.005324 15351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "003c7eeafd1646848d258201048b8147"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "285d2b73fa1958fb253067b0850c305f"
server_key_iv: "01cbd91d4321f72dce97091b8cee5577"
server_key_version: "encryptionkey@0"
I20260924 02:59:58.005625 15351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:58.021682 15351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:58.023849 15351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:58.023981 15351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:58.024195 15351 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:58.024474 15351 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:59:58.024505 15351 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:58.024539 15351 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:59:58.024562 15351 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:58.030227 15351 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:39559
I20260924 02:59:58.030267 15482 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:39559 every 8 connection(s)
I20260924 02:59:58.030628 15351 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 02:59:58.031579 15485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:58.031235 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:55937)
0924 02:59:58.031374 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:58.031441 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":72,"threads_started":1}
W20260924 02:59:58.031934 15483 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37233 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37233: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
I20260924 02:59:58.032153 15485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:58.032079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:49309)
0924 02:59:58.032115 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:58.032138 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 02:59:58.032339 15485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:58.032292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:52445)
0924 02:59:58.032312 (+ 20us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:58.032328 (+ 16us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11}
W20260924 02:59:58.032449 15483 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 02:59:58.037503 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 15351
I20260924 02:59:58.037732 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:38907
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37233
--webserver_port=33421
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 02:59:58.116755 15486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 02:59:58.116953 15486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 02:59:58.116974 15486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 02:59:58.118057 15486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 02:59:58.118088 15486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 02:59:58.118111 15486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 02:59:58.119644 15486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:38907
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33421
--tserver_master_addrs=127.0.98.62:37233
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15486
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 02:59:58.119818 15486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 02:59:58.119982 15486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 02:59:58.124603 15491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:58.124619 15492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 02:59:58.124610 15494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 02:59:58.124775 15486 server_base.cc:1034] running on GCE node
I20260924 02:59:58.124969 15486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 02:59:58.125172 15486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 02:59:58.126310 15486 hybrid_clock.cc:648] HybridClock initialized: now 1790218798126294 us; error 27 us; skew 500 ppm
Sep 24 02:59:58 dist-test-slave-sb6p krb5kdc[14052](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218798, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 02:59:58.135206 15486 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 02:59:58.135624 15486 webserver.cc:460] Webserver started at http://127.0.98.3:33421/ using document root <none> and password file <none>
I20260924 02:59:58.135797 15486 fs_manager.cc:359] Metadata directory not provided
I20260924 02:59:58.135835 15486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 02:59:58.137072 15486 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 02:59:58.137609 15501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:58.137790 15486 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:58.137871 15486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "d8c7d3c340174c888169ff5545d1787c"
format_stamp: "Formatted at 2026-09-24 02:59:56 on dist-test-slave-sb6p"
server_key: "fd2003ce1a11cc632108e1af0c634a45"
server_key_iv: "2f8fb8b5bdb5e78bc413d37ddb4337b7"
server_key_version: "encryptionkey@0"
I20260924 02:59:58.138214 15486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 02:59:58.148021 15486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 02:59:58.149613 15486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 02:59:58.149740 15486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 02:59:58.149930 15486 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 02:59:58.150185 15486 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 02:59:58.150215 15486 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:58.150235 15486 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 02:59:58.150259 15486 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 02:59:58.155472 15486 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:38907
I20260924 02:59:58.155539 15614 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:38907 every 8 connection(s)
I20260924 02:59:58.155820 15486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 02:59:58.156711 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:58.156374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:34717)
0924 02:59:58.156507 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:58.156570 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":70,"threads_started":1}
W20260924 02:59:58.157078 15615 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37233 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37233: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
I20260924 02:59:58.157317 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:58.157241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:42407)
0924 02:59:58.157279 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:58.157300 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 02:59:58.157553 15617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:58.157487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:48567)
0924 02:59:58.157519 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:58.157539 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 02:59:58.157637 15615 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 02:59:58.161820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 15486
I20260924 02:59:58.987586 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:58.987383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:48745)
0924 02:59:58.987500 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:58.987529 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":60,"threads_started":1}
I20260924 02:59:59.032884 15620 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:59.032694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:42133)
0924 02:59:59.032805 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:59.032838 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":57,"threads_started":1}
I20260924 02:59:59.137142 15621 raft_consensus.cc:482] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 02:59:59.137259 15621 raft_consensus.cc:504] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } } peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } }
I20260924 02:59:59.137523 15621 leader_election.cc:288] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907), 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559)
Sep 24 02:59:59 dist-test-slave-sb6p krb5kdc[14052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
Sep 24 02:59:59 dist-test-slave-sb6p krb5kdc[14052](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
I20260924 02:59:59.147327 15623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:59.137774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:54595 (local address 127.0.98.2:39559)
0924 02:59:59.137922 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:59.137931 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:59.137959 (+ 28us) server_negotiation.cc:407] Connection header received
0924 02:59:59.138397 (+ 438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:59.138399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:59.138435 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:59.138592 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:59.139280 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:59.140096 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:59.141095 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:59.141661 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:59.141777 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:59.144134 (+ 2357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:59.144138 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:59.144138 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:59.144150 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:59.145647 (+ 1497us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:59.146079 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:59.146081 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:59.146081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:59.146161 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:59.146395 (+ 234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:59.146395 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:59.146396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:59.146708 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:59.146777 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:59.147071 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 02:59:59.147217 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":50,"threads_started":1}
I20260924 02:59:59.147353 15622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:59.137663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:39559 (local address 127.0.98.1:54595)
0924 02:59:59.137808 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:59.137816 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:59.137852 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:59.138604 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:59.138607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:59.138986 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:59.139169 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:59.139175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:59.140184 (+ 1009us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:59.140185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:59.140993 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:59.140996 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:59.141749 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:59.141750 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:59.141873 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:59.142453 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:59.142457 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:59.144035 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:59.145724 (+ 1689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:59.145730 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:59.145732 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:59.146008 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:59.146244 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:59.146244 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:59.146245 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:59.146339 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:59.146772 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:59.146778 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:59.146919 (+ 141us) client_negotiation.cc:769] Sending connection context
0924 02:59:59.147018 (+ 99us) client_negotiation.cc:240] Negotiation successful
0924 02:59:59.147209 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":55,"thread_start_us":57,"threads_started":1}
I20260924 02:59:59.147395 15624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:59.137794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:41041 (local address 127.0.98.3:38907)
0924 02:59:59.137933 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0924 02:59:59.137941 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 02:59:59.137959 (+ 18us) server_negotiation.cc:407] Connection header received
0924 02:59:59.138359 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 02:59:59.138360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 02:59:59.138412 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 02:59:59.138557 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 02:59:59.139277 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:59.140126 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:59.141061 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 02:59:59.141662 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 02:59:59.141802 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:59.144478 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 02:59:59.144482 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 02:59:59.144483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 02:59:59.144495 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 02:59:59.146125 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:59.146411 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:59.146414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:59.146414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:59.146491 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 02:59:59.146695 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 02:59:59.146696 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 02:59:59.146696 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 02:59:59.146955 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 02:59:59.147034 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0924 02:59:59.147192 (+ 158us) server_negotiation.cc:299] Negotiation successful
0924 02:59:59.147313 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
I20260924 02:59:59.147347 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:59.137663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:38907 (local address 127.0.98.1:41041)
0924 02:59:59.137699 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:59.137708 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 02:59:59.137760 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 02:59:59.138604 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 02:59:59.138607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 02:59:59.138986 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
0924 02:59:59.139169 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:59.139175 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:59.140218 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:59.140219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:59.140992 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 02:59:59.140994 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 02:59:59.141781 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 02:59:59.141782 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 02:59:59.141874 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 02:59:59.142428 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 02:59:59.142438 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 02:59:59.144392 (+ 1954us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 02:59:59.146188 (+ 1796us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:59.146189 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:59.146190 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:59.146347 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:59.146576 (+ 229us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 02:59:59.146577 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 02:59:59.146577 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 02:59:59.146633 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 02:59:59.147020 (+ 387us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 02:59:59.147021 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 02:59:59.147064 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 02:59:59.147130 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 02:59:59.147233 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19,"mutex_wait_us":70}
W20260924 02:59:59.148478 15238 leader_election.cc:341] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559): Not found: Tablet not found: 849afbdd4a044004aa4b3ef8907b9acd
W20260924 02:59:59.148622 15235 leader_election.cc:341] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907): Not found: Tablet not found: 849afbdd4a044004aa4b3ef8907b9acd
I20260924 02:59:59.148665 15235 leader_election.cc:302] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa85f532f07f4f89bd994bc9b8849e47; no voters: 003c7eeafd1646848d258201048b8147, d8c7d3c340174c888169ff5545d1787c
I20260924 02:59:59.148772 15621 raft_consensus.cc:2726] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 02:59:59.158097 15625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:59.157895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:47825)
0924 02:59:59.158030 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:59.158064 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260924 02:59:59.988222 15626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 02:59:59.988015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:42719)
0924 02:59:59.988142 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 02:59:59.988178 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":60,"threads_started":1}
I20260924 03:00:00.033555 15627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:00.033342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:47667)
0924 03:00:00.033478 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:00.033516 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
I20260924 03:00:00.158771 15628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:00.158543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:48811)
0924 03:00:00.158678 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:00.158730 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":61,"threads_started":1}
I20260924 03:00:00.654405 15629 raft_consensus.cc:482] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:00:00.654498 15629 raft_consensus.cc:504] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } } peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } }
I20260924 03:00:00.654688 15629 leader_election.cc:288] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907), 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559)
W20260924 03:00:00.655093 15235 leader_election.cc:341] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907): Not found: Tablet not found: 849afbdd4a044004aa4b3ef8907b9acd
W20260924 03:00:00.655200 15238 leader_election.cc:341] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559): Not found: Tablet not found: 849afbdd4a044004aa4b3ef8907b9acd
I20260924 03:00:00.655256 15238 leader_election.cc:302] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa85f532f07f4f89bd994bc9b8849e47; no voters: 003c7eeafd1646848d258201048b8147, d8c7d3c340174c888169ff5545d1787c
I20260924 03:00:00.655323 15629 raft_consensus.cc:2726] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:00:00.988946 15630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:00.988692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:41617)
0924 03:00:00.988837 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:00.988882 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260924 03:00:01.034150 15631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:01.033966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:48883)
0924 03:00:01.034077 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:01.034110 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":60,"threads_started":1}
I20260924 03:00:01.159363 15632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:01.159150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:52761)
0924 03:00:01.159277 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:01.159319 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":72,"threads_started":1}
I20260924 03:00:01.989610 15633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:01.989406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:55867)
0924 03:00:01.989525 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:01.989561 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":58,"threads_started":1}
I20260924 03:00:02.034795 15634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:02.034581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:59217)
0924 03:00:02.034708 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:02.034756 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":55,"threads_started":1}
I20260924 03:00:02.160008 15635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:02.159798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:48257)
0924 03:00:02.159927 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:02.159967 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260924 03:00:02.936942 15636 raft_consensus.cc:482] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:00:02.937029 15636 raft_consensus.cc:504] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } } peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } }
I20260924 03:00:02.937163 15636 leader_election.cc:288] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907), 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559)
W20260924 03:00:02.937495 15235 leader_election.cc:341] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907): Not found: Tablet not found: 849afbdd4a044004aa4b3ef8907b9acd
W20260924 03:00:02.937578 15238 leader_election.cc:341] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559): Not found: Tablet not found: 849afbdd4a044004aa4b3ef8907b9acd
I20260924 03:00:02.937647 15238 leader_election.cc:302] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa85f532f07f4f89bd994bc9b8849e47; no voters: 003c7eeafd1646848d258201048b8147, d8c7d3c340174c888169ff5545d1787c
I20260924 03:00:02.937722 15636 raft_consensus.cc:2726] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:00:02.990330 15637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:02.990087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:40105)
0924 03:00:02.990235 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:02.990279 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":60,"threads_started":1}
I20260924 03:00:03.035404 15638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:03.035191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:38195)
0924 03:00:03.035326 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:03.035363 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260924 03:00:03.160676 15639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:03.160451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:41289)
0924 03:00:03.160583 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:03.160630 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260924 03:00:03.991154 15640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:03.990905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:40739)
0924 03:00:03.991062 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:03.991108 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":61,"threads_started":1}
I20260924 03:00:04.036116 15641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:04.035887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:40141)
0924 03:00:04.036026 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:04.036070 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":59,"threads_started":1}
I20260924 03:00:04.161268 15642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:04.161061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:35577)
0924 03:00:04.161185 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:04.161217 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":75,"threads_started":1}
I20260924 03:00:04.991829 15643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:04.991604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:43111)
0924 03:00:04.991741 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:04.991782 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260924 03:00:05.036806 15644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:05.036565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:40233)
0924 03:00:05.036715 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:05.036760 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":59,"threads_started":1}
I20260924 03:00:05.161935 15645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:05.161741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:55111)
0924 03:00:05.161854 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:05.161892 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260924 03:00:05.992504 15646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:05.992271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:43441)
0924 03:00:05.992412 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:05.992455 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":60,"threads_started":1}
I20260924 03:00:06.037437 15647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:06.037243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:35809)
0924 03:00:06.037359 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:06.037395 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260924 03:00:06.162508 15648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:06.162326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:46105)
0924 03:00:06.162432 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:06.162464 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":58,"threads_started":1}
I20260924 03:00:06.469790 15649 raft_consensus.cc:482] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:00:06.469887 15649 raft_consensus.cc:504] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } } peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } }
I20260924 03:00:06.470042 15649 leader_election.cc:288] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907), 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559)
W20260924 03:00:06.470448 15235 leader_election.cc:341] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907): Not found: Tablet not found: 849afbdd4a044004aa4b3ef8907b9acd
W20260924 03:00:06.470538 15238 leader_election.cc:341] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559): Not found: Tablet not found: 849afbdd4a044004aa4b3ef8907b9acd
I20260924 03:00:06.470604 15238 leader_election.cc:302] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa85f532f07f4f89bd994bc9b8849e47; no voters: 003c7eeafd1646848d258201048b8147, d8c7d3c340174c888169ff5545d1787c
I20260924 03:00:06.470664 15649 raft_consensus.cc:2726] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:00:06.993140 15650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:06.992902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:33379)
0924 03:00:06.993031 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:06.993086 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260924 03:00:07.038132 15651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:07.037908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:33363)
0924 03:00:07.038044 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:07.038087 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":60,"threads_started":1}
I20260924 03:00:07.163098 15652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:07.162900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:45673)
0924 03:00:07.163016 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:07.163055 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260924 03:00:07.993763 15653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:07.993561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:33375)
0924 03:00:07.993682 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:07.993717 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260924 03:00:08.038856 15654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:08.038659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:44553)
0924 03:00:08.038776 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:08.038812 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":60,"threads_started":1}
I20260924 03:00:08.163686 15655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:08.163484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:53519)
0924 03:00:08.163603 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:08.163641 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":70,"threads_started":1}
I20260924 03:00:08.994382 15656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:08.994191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:56067)
0924 03:00:08.994299 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:08.994334 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":53,"threads_started":1}
I20260924 03:00:09.039449 15657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:09.039256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:51695)
0924 03:00:09.039370 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:09.039408 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260924 03:00:09.164285 15658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:09.164096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:53461)
0924 03:00:09.164203 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:09.164239 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":59,"threads_started":1}
I20260924 03:00:09.995016 15659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:09.994817 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:53315)
0924 03:00:09.994929 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:09.994968 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260924 03:00:10.040105 15660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:10.039892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:51457)
0924 03:00:10.040012 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:10.040047 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260924 03:00:10.164935 15661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:10.164716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:51893)
0924 03:00:10.164860 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:10.164890 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":73,"threads_started":1}
I20260924 03:00:10.995625 15662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:10.995404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:41443)
0924 03:00:10.995543 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:10.995577 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":90,"threads_started":1}
I20260924 03:00:11.040794 15663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:11.040536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:59979)
0924 03:00:11.040697 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:11.040748 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":75,"threads_started":1}
I20260924 03:00:11.165602 15664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:11.165381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:51929)
0924 03:00:11.165515 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:11.165556 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":76,"threads_started":1}
I20260924 03:00:11.996271 15665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:11.996043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:55347)
0924 03:00:11.996178 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:11.996223 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":77,"threads_started":1}
I20260924 03:00:12.041500 15666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:12.041261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:34367)
0924 03:00:12.041409 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:12.041454 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260924 03:00:12.166304 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:12.166096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:43449)
0924 03:00:12.166214 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:12.166258 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":57,"threads_started":1}
I20260924 03:00:12.182447 15668 raft_consensus.cc:482] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:00:12.182564 15668 raft_consensus.cc:504] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8c7d3c340174c888169ff5545d1787c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 38907 } } peers { permanent_uuid: "fa85f532f07f4f89bd994bc9b8849e47" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 37619 } } peers { permanent_uuid: "003c7eeafd1646848d258201048b8147" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 39559 } }
I20260924 03:00:12.182734 15668 leader_election.cc:288] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907), 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559)
W20260924 03:00:12.183096 15235 leader_election.cc:341] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer d8c7d3c340174c888169ff5545d1787c (127.0.98.3:38907): Not found: Tablet not found: 849afbdd4a044004aa4b3ef8907b9acd
W20260924 03:00:12.183156 15238 leader_election.cc:341] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 003c7eeafd1646848d258201048b8147 (127.0.98.2:39559): Not found: Tablet not found: 849afbdd4a044004aa4b3ef8907b9acd
I20260924 03:00:12.183224 15238 leader_election.cc:302] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fa85f532f07f4f89bd994bc9b8849e47; no voters: 003c7eeafd1646848d258201048b8147, d8c7d3c340174c888169ff5545d1787c
I20260924 03:00:12.183310 15668 raft_consensus.cc:2726] T 849afbdd4a044004aa4b3ef8907b9acd P fa85f532f07f4f89bd994bc9b8849e47 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:00:12.996933 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:12.996721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.1:34325)
0924 03:00:12.996854 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:12.996887 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260924 03:00:13.042230 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:13.042023 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.2:41575)
0924 03:00:13.042138 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:13.042185 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":55,"threads_started":1}
I20260924 03:00:13.166987 15671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:13.166789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37233 (local address 127.0.98.3:32871)
0924 03:00:13.166904 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:13.166941 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37233: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260924 03:00:13.167950 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 15190
I20260924 03:00:13.172554 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 15351
I20260924 03:00:13.176082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 15486
2026-09-24T03:00:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18092 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:00:13 dist-test-slave-sb6p krb5kdc[15674](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:00:13 dist-test-slave-sb6p krb5kdc[15674](info): set up 2 sockets
Sep 24 03:00:13 dist-test-slave-sb6p krb5kdc[15674](info): commencing operation
krb5kdc: starting...
W20260924 03:00:14.122246 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:00:14 dist-test-slave-sb6p krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:00:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:00:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:00:14.233564 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37721
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:00:14.311756 15687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:00:14.311959 15687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:00:14.311990 15687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:00:14.313117 15687 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:00:14.313153 15687 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:00:14.313169 15687 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:00:14.313182 15687 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:00:14.313198 15687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:00:14.313215 15687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:00:14.313235 15687 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:00:14.313258 15687 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:00:14.314725 15687 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15687
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:00:14.314961 15687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:00:14.315169 15687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:00:14.320027 15692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:14.320039 15695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:14.320031 15693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:00:14.320057 15687 server_base.cc:1034] running on GCE node
I20260924 03:00:14.320458 15687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:00:14.320775 15687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:00:14.321939 15687 hybrid_clock.cc:648] HybridClock initialized: now 1790218814321920 us; error 29 us; skew 500 ppm
Sep 24 03:00:14 dist-test-slave-sb6p krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:00:14.324630 15687 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:00:14.325064 15687 webserver.cc:460] Webserver started at http://127.0.98.62:37251/ using document root <none> and password file <none>
I20260924 03:00:14.325291 15687 fs_manager.cc:359] Metadata directory not provided
I20260924 03:00:14.325340 15687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:00:14.325462 15687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:00:14.326555 15687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "0d379a44b74d40919e53fd3d056209e2"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "efe780b253f01f60ae97580cfcfe4151"
server_key_iv: "3a6967d827c111046a13f99ec9fc7a12"
server_key_version: "encryptionkey@0"
I20260924 03:00:14.326898 15687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "0d379a44b74d40919e53fd3d056209e2"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "efe780b253f01f60ae97580cfcfe4151"
server_key_iv: "3a6967d827c111046a13f99ec9fc7a12"
server_key_version: "encryptionkey@0"
I20260924 03:00:14.328116 15687 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:00:14.328804 15702 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:14.328994 15687 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:00:14.329073 15687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "0d379a44b74d40919e53fd3d056209e2"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "efe780b253f01f60ae97580cfcfe4151"
server_key_iv: "3a6967d827c111046a13f99ec9fc7a12"
server_key_version: "encryptionkey@0"
I20260924 03:00:14.329126 15687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:00:14.340557 15687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:00:14.342612 15687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:00:14.342720 15687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:00:14.346158 15687 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37721
I20260924 03:00:14.346203 15754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37721 every 8 connection(s)
I20260924 03:00:14.346506 15687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:00:14.347337 15755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:00:14.347635 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 15687
I20260924 03:00:14.347733 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:00:14.347913 392 external_mini_cluster.cc:1428] Setting key c5cdaa9879da354a84bd7226d6d46b7b
I20260924 03:00:14.350180 15755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: Bootstrap starting.
I20260924 03:00:14.350875 15755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: Neither blocks nor log segments found. Creating new log.
I20260924 03:00:14.351142 15755 log.cc:824] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:00:14.351857 15755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: No bootstrap required, opened a new log
I20260924 03:00:14.354103 15755 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } }
I20260924 03:00:14.354271 15755 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:00:14.354296 15755 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d379a44b74d40919e53fd3d056209e2, State: Initialized, Role: FOLLOWER
I20260924 03:00:14.354413 15755 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } }
I20260924 03:00:14.354483 15755 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:00:14.354506 15755 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:00:14.354574 15755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 0 FOLLOWER]: Advancing to term 1
Sep 24 03:00:14 dist-test-slave-sb6p krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:00:14.355227 15755 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } }
I20260924 03:00:14.355320 15755 leader_election.cc:302] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d379a44b74d40919e53fd3d056209e2; no voters:
I20260924 03:00:14.355468 15755 leader_election.cc:288] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:00:14.355619 15761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:00:14.355738 15755 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:00:14.355782 15761 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 1 LEADER]: Becoming Leader. State: Replica: 0d379a44b74d40919e53fd3d056209e2, State: Running, Role: LEADER
I20260924 03:00:14.355890 15761 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } }
I20260924 03:00:14.356282 15761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d379a44b74d40919e53fd3d056209e2. Latest consensus state: current_term: 1 leader_uuid: "0d379a44b74d40919e53fd3d056209e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } } }
I20260924 03:00:14.356298 15760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d379a44b74d40919e53fd3d056209e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } } }
I20260924 03:00:14.356451 15760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:00:14.356650 15761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:00:14.356981 15769 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:00:14.357525 15769 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:00:14.358703 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.348380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45512 (local address 127.0.98.62:37721)
0924 03:00:14.348561 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:14.348571 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:14.348605 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:00:14.349106 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:14.349115 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:14.349176 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:14.349321 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:00:14.349881 (+ 560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.351187 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.352163 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.352562 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.352641 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.355167 (+ 2526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:00:14.355171 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:00:14.355172 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:00:14.355182 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:00:14.356950 (+ 1768us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:14.357425 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:14.357428 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:14.357428 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:14.357529 (+ 101us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:14.357795 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:14.357797 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:14.357797 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:14.358082 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:00:14.358242 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:14.358361 (+ 119us) server_negotiation.cc:299] Negotiation successful
0924 03:00:14.358492 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":70,"threads_started":1}
I20260924 03:00:14.359129 15769 catalog_manager.cc:1347] Generated new cluster ID: f9323299bc7348899ed998b3f66b531e
I20260924 03:00:14.359179 15769 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:00:14.371239 15769 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:00:14.371774 15769 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:00:14.376286 15769 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: Generated new TSK 0
I20260924 03:00:14.376454 15769 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:00:14.405973 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37721
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:00:14.490770 15781 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:00:14.490985 15781 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:00:14.491010 15781 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:00:14.492185 15781 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:00:14.492228 15781 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:00:14.492257 15781 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:00:14.493716 15781 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15781
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:00:14.493940 15781 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:00:14.494189 15781 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:00:14.499076 15787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:00:14.499135 15781 server_base.cc:1034] running on GCE node
W20260924 03:00:14.499085 15789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:14.499104 15786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:00:14.499505 15781 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:00:14.499743 15781 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:00:14.500893 15781 hybrid_clock.cc:648] HybridClock initialized: now 1790218814500877 us; error 28 us; skew 500 ppm
Sep 24 03:00:14 dist-test-slave-sb6p krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:00:14.503443 15781 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:00:14.503855 15781 webserver.cc:460] Webserver started at http://127.0.98.1:35663/ using document root <none> and password file <none>
I20260924 03:00:14.504047 15781 fs_manager.cc:359] Metadata directory not provided
I20260924 03:00:14.504094 15781 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:00:14.504238 15781 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:00:14.505298 15781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "9992277a03de478d9bcccf4c57b0629c"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "9ec74d6137abcdab004bc7f00cb7a8fb"
server_key_iv: "4702dbb084f8b37ba8d54b5f67213d5e"
server_key_version: "encryptionkey@0"
I20260924 03:00:14.505622 15781 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "9992277a03de478d9bcccf4c57b0629c"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "9ec74d6137abcdab004bc7f00cb7a8fb"
server_key_iv: "4702dbb084f8b37ba8d54b5f67213d5e"
server_key_version: "encryptionkey@0"
I20260924 03:00:14.506925 15781 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260924 03:00:14.507668 15796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:14.507850 15781 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:00:14.507932 15781 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "9992277a03de478d9bcccf4c57b0629c"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "9ec74d6137abcdab004bc7f00cb7a8fb"
server_key_iv: "4702dbb084f8b37ba8d54b5f67213d5e"
server_key_version: "encryptionkey@0"
I20260924 03:00:14.507993 15781 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:00:14.521165 15781 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:00:14.522887 15781 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:00:14.523016 15781 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:00:14.523217 15781 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:00:14.523489 15781 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:00:14.523527 15781 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:14.523555 15781 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:00:14.523573 15781 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:14.529016 15781 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38161
I20260924 03:00:14.529074 15909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38161 every 8 connection(s)
I20260924 03:00:14.529362 15781 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:00:14.530959 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 15781
I20260924 03:00:14.531045 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:00:14.531239 392 external_mini_cluster.cc:1428] Setting key b4ed674b1d81e7812a61edda269d82d1
Sep 24 03:00:14 dist-test-slave-sb6p krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:00:14.538954 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.530040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57319 (local address 127.0.98.62:37721)
0924 03:00:14.530101 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:14.530103 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:14.530279 (+ 176us) server_negotiation.cc:407] Connection header received
0924 03:00:14.530790 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:14.530791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:14.530803 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:14.530867 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:00:14.531949 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.532520 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.533690 (+ 1170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.534010 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.534079 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.536384 (+ 2305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:00:14.536385 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:00:14.536386 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:00:14.536393 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:00:14.537602 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:14.538040 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:14.538040 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:14.538041 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:14.538084 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:14.538334 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:14.538335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:14.538335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:14.538426 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:00:14.538467 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:14.538812 (+ 345us) server_negotiation.cc:299] Negotiation successful
0924 03:00:14.538887 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 03:00:14.539192 15912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.530015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:57319)
0924 03:00:14.530163 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:14.530200 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:14.530704 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:14.530925 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:14.530929 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:14.531602 (+ 673us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:00:14.531862 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:14.531870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.532605 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.532605 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:14.533608 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:14.533610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.534088 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.534088 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:14.534186 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.534676 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:00:14.534684 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:00:14.536294 (+ 1610us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:00:14.537677 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:14.537684 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:14.537687 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:14.537976 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:14.538139 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:14.538139 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:14.538140 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:14.538271 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:14.538481 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:00:14.538487 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:00:14.538678 (+ 191us) client_negotiation.cc:769] Sending connection context
0924 03:00:14.538797 (+ 119us) client_negotiation.cc:240] Negotiation successful
0924 03:00:14.538965 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":84,"threads_started":1}
I20260924 03:00:14.539759 15910 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37721
I20260924 03:00:14.539855 15910 heartbeater.cc:464] Registering TS with master...
I20260924 03:00:14.540026 15910 heartbeater.cc:511] Master 127.0.98.62:37721 requested a full tablet report, sending...
I20260924 03:00:14.540454 15719 ts_manager.cc:194] Registered new tserver with Master: 9992277a03de478d9bcccf4c57b0629c (127.0.98.1:38161)
I20260924 03:00:14.541025 15719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:57319
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:00:14.553403 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37721
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:00:14.632965 15915 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:00:14.633162 15915 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:00:14.633183 15915 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:00:14.634316 15915 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:00:14.634353 15915 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:00:14.634383 15915 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:00:14.635799 15915 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15915
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:00:14.635980 15915 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:00:14.636160 15915 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:00:14.640950 15923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:00:14.641004 15915 server_base.cc:1034] running on GCE node
W20260924 03:00:14.640966 15921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:14.640955 15920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:00:14.641324 15915 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:00:14.641549 15915 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:00:14.642719 15915 hybrid_clock.cc:648] HybridClock initialized: now 1790218814642703 us; error 31 us; skew 500 ppm
Sep 24 03:00:14 dist-test-slave-sb6p krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:00:14.645053 15915 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:00:14.645462 15915 webserver.cc:460] Webserver started at http://127.0.98.2:38423/ using document root <none> and password file <none>
I20260924 03:00:14.645678 15915 fs_manager.cc:359] Metadata directory not provided
I20260924 03:00:14.645722 15915 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:00:14.645927 15915 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:00:14.646996 15915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "18034413df9f6b211b599017e9c07054"
server_key_iv: "2956a93c75b5c4c412772e391cc96500"
server_key_version: "encryptionkey@0"
I20260924 03:00:14.647346 15915 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "18034413df9f6b211b599017e9c07054"
server_key_iv: "2956a93c75b5c4c412772e391cc96500"
server_key_version: "encryptionkey@0"
I20260924 03:00:14.648622 15915 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:00:14.649430 15930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:14.649598 15915 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:00:14.649677 15915 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "18034413df9f6b211b599017e9c07054"
server_key_iv: "2956a93c75b5c4c412772e391cc96500"
server_key_version: "encryptionkey@0"
I20260924 03:00:14.649746 15915 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:00:14.664685 15915 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:00:14.666720 15915 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:00:14.666842 15915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:00:14.667070 15915 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:00:14.667361 15915 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:00:14.667394 15915 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:14.667423 15915 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:00:14.667445 15915 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:14.672663 15915 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41077
I20260924 03:00:14.672720 16043 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41077 every 8 connection(s)
I20260924 03:00:14.673004 15915 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:00:14.677728 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 15915
I20260924 03:00:14.677822 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:00:14.678001 392 external_mini_cluster.cc:1428] Setting key 32296e39f5b5410b3173ba3dc3ea5a7e
Sep 24 03:00:14 dist-test-slave-sb6p krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:00:14.681715 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.673663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:53803 (local address 127.0.98.62:37721)
0924 03:00:14.673736 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:14.673739 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:14.673922 (+ 183us) server_negotiation.cc:407] Connection header received
0924 03:00:14.674382 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:14.674383 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:14.674394 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:14.674462 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:00:14.675349 (+ 887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.675860 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.676773 (+ 913us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.676950 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.676993 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.679093 (+ 2100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:00:14.679094 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:00:14.679094 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:00:14.679101 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:00:14.680525 (+ 1424us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:14.680884 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:14.680885 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:14.680886 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:14.680949 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:14.681136 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:14.681137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:14.681137 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:14.681268 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:00:14.681326 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:14.681555 (+ 229us) server_negotiation.cc:299] Negotiation successful
0924 03:00:14.681628 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260924 03:00:14.681799 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.673641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:53803)
0924 03:00:14.673805 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:14.673839 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:14.674306 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:14.674549 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:14.674553 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:14.675087 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:00:14.675273 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:14.675279 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.675923 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.675923 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:14.676716 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:14.676718 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.677000 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.677001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:14.677090 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.677495 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:00:14.677502 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:00:14.678992 (+ 1490us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:00:14.680610 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:14.680615 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:14.680617 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:14.680815 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:14.681009 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:14.681009 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:14.681009 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:14.681090 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:14.681323 (+ 233us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:00:14.681325 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:00:14.681450 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 03:00:14.681524 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 03:00:14.681636 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260924 03:00:14.682216 16044 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37721
I20260924 03:00:14.682302 16044 heartbeater.cc:464] Registering TS with master...
I20260924 03:00:14.682476 16044 heartbeater.cc:511] Master 127.0.98.62:37721 requested a full tablet report, sending...
I20260924 03:00:14.682880 15719 ts_manager.cc:194] Registered new tserver with Master: 69ed3bcd3eea43fbb1cedc5d3776bc39 (127.0.98.2:41077)
I20260924 03:00:14.683379 15719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:53803
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:00:14.701287 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37721
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:00:14.779277 16049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:00:14.779481 16049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:00:14.779505 16049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:00:14.780628 16049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:00:14.780673 16049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:00:14.780699 16049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:00:14.782091 16049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16049
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:00:14.782270 16049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:00:14.782439 16049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:00:14.787084 16054 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:14.787091 16055 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:14.787089 16057 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:00:14.787279 16049 server_base.cc:1034] running on GCE node
I20260924 03:00:14.787485 16049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:00:14.787705 16049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:00:14.788844 16049 hybrid_clock.cc:648] HybridClock initialized: now 1790218814788824 us; error 29 us; skew 500 ppm
Sep 24 03:00:14 dist-test-slave-sb6p krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:00:14.791160 16049 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:00:14.791569 16049 webserver.cc:460] Webserver started at http://127.0.98.3:33441/ using document root <none> and password file <none>
I20260924 03:00:14.791759 16049 fs_manager.cc:359] Metadata directory not provided
I20260924 03:00:14.791805 16049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:00:14.791921 16049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:00:14.792927 16049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "237e991937424c92b84adf2579f7a2fa"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "2ea8fa57b029cfc439ff0614a8bd9c1f"
server_key_iv: "8a887b272cb211ef390d0ded6ce2164f"
server_key_version: "encryptionkey@0"
I20260924 03:00:14.793241 16049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "237e991937424c92b84adf2579f7a2fa"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "2ea8fa57b029cfc439ff0614a8bd9c1f"
server_key_iv: "8a887b272cb211ef390d0ded6ce2164f"
server_key_version: "encryptionkey@0"
I20260924 03:00:14.794423 16049 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260924 03:00:14.795145 16064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:14.795286 16049 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:14.795364 16049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "237e991937424c92b84adf2579f7a2fa"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "2ea8fa57b029cfc439ff0614a8bd9c1f"
server_key_iv: "8a887b272cb211ef390d0ded6ce2164f"
server_key_version: "encryptionkey@0"
I20260924 03:00:14.795430 16049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:00:14.808390 16049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:00:14.810439 16049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:00:14.810626 16049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:00:14.810853 16049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:00:14.811136 16049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:00:14.811167 16049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:14.811197 16049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:00:14.811220 16049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:14.816571 16049 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35361
I20260924 03:00:14.816628 16177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35361 every 8 connection(s)
I20260924 03:00:14.816949 16049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
Sep 24 03:00:14 dist-test-slave-sb6p krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:00:14.825220 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.817596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:37133 (local address 127.0.98.62:37721)
0924 03:00:14.817644 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:14.817646 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:14.817886 (+ 240us) server_negotiation.cc:407] Connection header received
0924 03:00:14.818379 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:14.818381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:14.818393 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:14.818441 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:00:14.819394 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.819891 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.820843 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.821050 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.821104 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.823161 (+ 2057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:00:14.823162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:00:14.823162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:00:14.823169 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:00:14.824221 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:14.824549 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:14.824550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:14.824550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:14.824590 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:14.824757 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:14.824757 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:14.824757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:14.824850 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:00:14.824896 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:14.825099 (+ 203us) server_negotiation.cc:299] Negotiation successful
0924 03:00:14.825156 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:00:14.825376 16180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.817559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:37133)
0924 03:00:14.817745 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:14.817792 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:14.818278 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:14.818550 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:14.818553 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:14.819097 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:00:14.819292 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:14.819296 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.819970 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.819971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:14.820766 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:14.820768 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.821117 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.821118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:14.821230 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.821670 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:00:14.821679 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:00:14.823078 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:00:14.824285 (+ 1207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:14.824291 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:14.824292 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:14.824495 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:14.824632 (+ 137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:14.824633 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:14.824633 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:14.824718 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:14.824892 (+ 174us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:00:14.824894 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:00:14.825005 (+ 111us) client_negotiation.cc:769] Sending connection context
0924 03:00:14.825082 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 03:00:14.825194 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":67,"threads_started":1}
I20260924 03:00:14.825641 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 16049
I20260924 03:00:14.825731 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:00:14.825835 16178 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37721
I20260924 03:00:14.825932 16178 heartbeater.cc:464] Registering TS with master...
I20260924 03:00:14.825948 392 external_mini_cluster.cc:1428] Setting key 0482d07d9a03e5ee13d52c3e8297b635
I20260924 03:00:14.826097 16178 heartbeater.cc:511] Master 127.0.98.62:37721 requested a full tablet report, sending...
I20260924 03:00:14.826429 15719 ts_manager.cc:194] Registered new tserver with Master: 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361)
I20260924 03:00:14.826763 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:00:14.826905 15719 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:37133
I20260924 03:00:14.833047 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.828022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45522 (local address 127.0.98.62:37721)
0924 03:00:14.828059 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:14.828061 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:14.828158 (+ 97us) server_negotiation.cc:407] Connection header received
0924 03:00:14.828204 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:14.828204 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:14.828212 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:14.828251 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:00:14.828719 (+ 468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.829163 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.829829 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.830005 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.830044 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.831052 (+ 1008us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:00:14.831053 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:00:14.831053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:00:14.831058 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:00:14.832092 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:14.832396 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:14.832396 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:14.832396 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:14.832442 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:14.832573 (+ 131us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:14.832574 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:14.832574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:14.832690 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:00:14.832761 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:14.832887 (+ 126us) server_negotiation.cc:299] Negotiation successful
0924 03:00:14.832962 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:00:14.834335 15719 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45522:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:00:14.835924 15719 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:00:14.841531 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.837761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41077 (local address 127.0.0.1:54700)
0924 03:00:14.838331 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:14.838339 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:14.838383 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:14.838622 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:14.838622 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:14.838625 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:00:14.838661 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:14.838662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.839843 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.839844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:14.840658 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:14.840661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.841258 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.841260 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:14.841331 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.841345 (+ 14us) client_negotiation.cc:769] Sending connection context
0924 03:00:14.841408 (+ 63us) client_negotiation.cc:240] Negotiation successful
0924 03:00:14.841455 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":557,"mutex_wait_us":18,"thread_start_us":70,"threads_started":1}
I20260924 03:00:14.841545 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.837860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54700 (local address 127.0.98.2:41077)
0924 03:00:14.838007 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:14.838012 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:14.838388 (+ 376us) server_negotiation.cc:407] Connection header received
0924 03:00:14.838438 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:14.838440 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:14.838455 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:14.838544 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:00:14.838740 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.839762 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.840717 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.841188 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.841249 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.841307 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:14.841428 (+ 121us) server_negotiation.cc:299] Negotiation successful
0924 03:00:14.841466 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260924 03:00:14.841878 16186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.837761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35361 (local address 127.0.0.1:36316)
0924 03:00:14.838026 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:14.838064 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:14.838126 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:14.838292 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:14.838295 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:14.838297 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:00:14.838354 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:14.838363 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.839663 (+ 1300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.839664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:14.840712 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:14.840714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.841645 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.841647 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:14.841710 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.841714 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:00:14.841756 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 03:00:14.841791 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":47,"threads_started":1}
I20260924 03:00:14.841879 16187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.837762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38161 (local address 127.0.0.1:50106)
0924 03:00:14.838191 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:14.838199 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:14.838252 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:14.838466 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:14.838466 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:14.838468 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:00:14.838513 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:14.838535 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.839428 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.839429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:14.840682 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:14.840685 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.841700 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.841701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:14.841748 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.841751 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:00:14.841786 (+ 35us) client_negotiation.cc:240] Negotiation successful
0924 03:00:14.841813 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":419,"mutex_wait_us":23,"thread_start_us":70,"threads_started":1}
I20260924 03:00:14.842104 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.837969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36316 (local address 127.0.98.3:35361)
0924 03:00:14.838107 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:14.838110 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:14.838119 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:00:14.838204 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:14.838206 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:14.838221 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:14.838285 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:00:14.838436 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.839570 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.840790 (+ 1220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.841493 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.841559 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.841609 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:14.841987 (+ 378us) server_negotiation.cc:299] Negotiation successful
0924 03:00:14.842034 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260924 03:00:14.842321 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.837992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50106 (local address 127.0.98.1:38161)
0924 03:00:14.838156 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:14.838160 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:14.838261 (+ 101us) server_negotiation.cc:407] Connection header received
0924 03:00:14.838311 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:14.838313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:14.838329 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:14.838399 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:00:14.838615 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.839341 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.840777 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.841496 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.841559 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.842176 (+ 617us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:14.842201 (+ 25us) server_negotiation.cc:299] Negotiation successful
0924 03:00:14.842241 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":74,"threads_started":1}
I20260924 03:00:14.842209 15978 tablet_service.cc:1467] Processing CreateTablet for tablet d2b24f2afdce4c229cb3eed174ae4328 (DEFAULT_TABLE table=test-table [id=681f8a35551a4bcd9b0499af1bbe82c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:00:14.842581 15978 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b24f2afdce4c229cb3eed174ae4328. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:00:14.842905 15844 tablet_service.cc:1467] Processing CreateTablet for tablet d2b24f2afdce4c229cb3eed174ae4328 (DEFAULT_TABLE table=test-table [id=681f8a35551a4bcd9b0499af1bbe82c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:00:14.843183 15844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b24f2afdce4c229cb3eed174ae4328. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:00:14.843137 16112 tablet_service.cc:1467] Processing CreateTablet for tablet d2b24f2afdce4c229cb3eed174ae4328 (DEFAULT_TABLE table=test-table [id=681f8a35551a4bcd9b0499af1bbe82c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:00:14.843429 16112 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2b24f2afdce4c229cb3eed174ae4328. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:00:14.845522 16192 tablet_bootstrap.cc:492] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c: Bootstrap starting.
I20260924 03:00:14.845822 16193 tablet_bootstrap.cc:492] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39: Bootstrap starting.
I20260924 03:00:14.846311 16192 tablet_bootstrap.cc:654] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c: Neither blocks nor log segments found. Creating new log.
I20260924 03:00:14.846400 16193 tablet_bootstrap.cc:654] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39: Neither blocks nor log segments found. Creating new log.
I20260924 03:00:14.846376 16194 tablet_bootstrap.cc:492] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa: Bootstrap starting.
I20260924 03:00:14.846938 16194 tablet_bootstrap.cc:654] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa: Neither blocks nor log segments found. Creating new log.
I20260924 03:00:14.846992 16193 log.cc:824] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39: Log is configured to *not* fsync() on all Append() calls
I20260924 03:00:14.847074 16192 log.cc:824] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:00:14.847435 16194 log.cc:824] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa: Log is configured to *not* fsync() on all Append() calls
I20260924 03:00:14.847724 16192 tablet_bootstrap.cc:492] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c: No bootstrap required, opened a new log
I20260924 03:00:14.847724 16193 tablet_bootstrap.cc:492] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39: No bootstrap required, opened a new log
I20260924 03:00:14.847793 16192 ts_tablet_manager.cc:1397] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:00:14.847793 16193 ts_tablet_manager.cc:1397] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260924 03:00:14.848073 16194 tablet_bootstrap.cc:492] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa: No bootstrap required, opened a new log
I20260924 03:00:14.848129 16194 ts_tablet_manager.cc:1397] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:00:14.849644 16194 raft_consensus.cc:348] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:14.849648 16193 raft_consensus.cc:348] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:14.849648 16192 raft_consensus.cc:348] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:14.849777 16194 raft_consensus.cc:374] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:00:14.849802 16192 raft_consensus.cc:374] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:00:14.849808 16194 raft_consensus.cc:729] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 237e991937424c92b84adf2579f7a2fa, State: Initialized, Role: FOLLOWER
I20260924 03:00:14.849813 16193 raft_consensus.cc:374] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:00:14.849846 16193 raft_consensus.cc:729] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ed3bcd3eea43fbb1cedc5d3776bc39, State: Initialized, Role: FOLLOWER
I20260924 03:00:14.849846 16192 raft_consensus.cc:729] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9992277a03de478d9bcccf4c57b0629c, State: Initialized, Role: FOLLOWER
I20260924 03:00:14.849948 16193 consensus_queue.cc:260] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:14.849948 16192 consensus_queue.cc:260] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:14.849989 16194 consensus_queue.cc:260] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:14.850194 16193 ts_tablet_manager.cc:1428] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 03:00:14.850194 16192 ts_tablet_manager.cc:1428] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:00:14.850320 15910 heartbeater.cc:503] Master 127.0.98.62:37721 was elected leader, sending a full tablet report...
I20260924 03:00:14.850317 16044 heartbeater.cc:503] Master 127.0.98.62:37721 was elected leader, sending a full tablet report...
I20260924 03:00:14.850268 16194 ts_tablet_manager.cc:1428] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:00:14.850322 16178 heartbeater.cc:503] Master 127.0.98.62:37721 was elected leader, sending a full tablet report...
I20260924 03:00:14.902793 16199 raft_consensus.cc:482] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:00:14.902925 16199 raft_consensus.cc:504] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:14.903280 16199 leader_election.cc:288] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361), 9992277a03de478d9bcccf4c57b0629c (127.0.98.1:38161)
I20260924 03:00:14.906083 16191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.903516 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:32933 (local address 127.0.98.1:38161)
0924 03:00:14.903560 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:14.903562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:14.903714 (+ 152us) server_negotiation.cc:407] Connection header received
0924 03:00:14.903746 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:14.903747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:14.903760 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:14.903835 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:00:14.903972 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.904418 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.905402 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.905772 (+ 370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.905837 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.905847 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:14.905978 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 03:00:14.906015 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:00:14.906092 16190 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.903524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:47955 (local address 127.0.98.3:35361)
0924 03:00:14.903572 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:14.903573 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:14.903582 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:00:14.903616 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:14.903617 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:14.903629 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:14.903696 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:00:14.903888 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.904422 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.905415 (+ 993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.905775 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.905834 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.905844 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:14.905992 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 03:00:14.906032 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 03:00:14.906090 16046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.903457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35361 (local address 127.0.98.2:47955)
0924 03:00:14.903516 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:14.903527 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:14.903568 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:14.903712 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:14.903713 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:14.903715 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:00:14.903791 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:14.903792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.904501 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.904502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:14.905326 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:14.905329 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.905849 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.905850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:14.905925 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.905928 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:00:14.905974 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 03:00:14.906015 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":45}
I20260924 03:00:14.906100 16201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.903457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38161 (local address 127.0.98.2:32933)
0924 03:00:14.903636 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:14.903646 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:14.903703 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:14.903838 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:14.903839 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:14.903840 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:00:14.903912 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:14.903914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.904485 (+ 571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.904486 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:14.905313 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:14.905315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.905845 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.905846 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:14.905914 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.905917 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:00:14.905958 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 03:00:14.905998 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":43,"thread_start_us":96,"threads_started":1}
I20260924 03:00:14.906441 16132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b24f2afdce4c229cb3eed174ae4328" candidate_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "237e991937424c92b84adf2579f7a2fa" is_pre_election: true
I20260924 03:00:14.906553 15864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b24f2afdce4c229cb3eed174ae4328" candidate_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9992277a03de478d9bcccf4c57b0629c" is_pre_election: true
I20260924 03:00:14.906611 16132 raft_consensus.cc:2454] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ed3bcd3eea43fbb1cedc5d3776bc39 in term 0.
I20260924 03:00:14.906700 15864 raft_consensus.cc:2454] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69ed3bcd3eea43fbb1cedc5d3776bc39 in term 0.
I20260924 03:00:14.906822 15932 leader_election.cc:302] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 237e991937424c92b84adf2579f7a2fa, 69ed3bcd3eea43fbb1cedc5d3776bc39; no voters:
I20260924 03:00:14.906981 16199 raft_consensus.cc:2781] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:00:14.907023 16199 raft_consensus.cc:482] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:00:14.907037 16199 raft_consensus.cc:3037] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:00:14.907737 16199 raft_consensus.cc:504] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:14.907860 16199 leader_election.cc:288] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [CANDIDATE]: Term 1 election: Requested vote from peers 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361), 9992277a03de478d9bcccf4c57b0629c (127.0.98.1:38161)
I20260924 03:00:14.908002 15864 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b24f2afdce4c229cb3eed174ae4328" candidate_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9992277a03de478d9bcccf4c57b0629c"
I20260924 03:00:14.908030 16132 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2b24f2afdce4c229cb3eed174ae4328" candidate_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "237e991937424c92b84adf2579f7a2fa"
I20260924 03:00:14.908085 15864 raft_consensus.cc:3037] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:00:14.908097 16132 raft_consensus.cc:3037] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:00:14.908738 16132 raft_consensus.cc:2454] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ed3bcd3eea43fbb1cedc5d3776bc39 in term 1.
I20260924 03:00:14.908746 15864 raft_consensus.cc:2454] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69ed3bcd3eea43fbb1cedc5d3776bc39 in term 1.
I20260924 03:00:14.908886 15932 leader_election.cc:302] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 237e991937424c92b84adf2579f7a2fa, 69ed3bcd3eea43fbb1cedc5d3776bc39; no voters:
I20260924 03:00:14.908977 16199 raft_consensus.cc:2781] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:00:14.909116 16199 raft_consensus.cc:686] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 1 LEADER]: Becoming Leader. State: Replica: 69ed3bcd3eea43fbb1cedc5d3776bc39, State: Running, Role: LEADER
I20260924 03:00:14.909198 16199 consensus_queue.cc:237] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:14.909956 15719 catalog_manager.cc:5495] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69ed3bcd3eea43fbb1cedc5d3776bc39 (127.0.98.2). New cstate: current_term: 1 leader_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } health_report { overall_health: UNKNOWN } } }
I20260924 03:00:14.916287 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.914391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54702 (local address 127.0.98.2:41077)
0924 03:00:14.914436 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:14.914438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:14.914455 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:00:14.914479 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:14.914480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:14.914495 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:14.914567 (+ 72us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:00:14.914659 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.915089 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.915749 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:14.915924 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:14.915971 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:14.916068 (+ 97us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:00:14.916151 (+ 83us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:00:14.916203 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:14.916215 (+ 12us) server_negotiation.cc:299] Negotiation successful
0924 03:00:14.916229 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 03:00:14.917796 16132 raft_consensus.cc:1261] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa [term 1 FOLLOWER]: Refusing update from remote peer 69ed3bcd3eea43fbb1cedc5d3776bc39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:00:14.917796 15864 raft_consensus.cc:1261] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Refusing update from remote peer 69ed3bcd3eea43fbb1cedc5d3776bc39: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:00:14.918082 16202 consensus_queue.cc:1035] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:00:14.918196 16199 consensus_queue.cc:1035] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [LEADER]: Connected to new peer: Peer: permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:00:14.920262 16205 mvcc.cc:205] Tried to move back new op lower bound from 7332736265901199360 to 7332736265868636160. Current Snapshot: MvccSnapshot[applied={T|T < 7332736265901199360}]
I20260924 03:00:14.920361 16207 mvcc.cc:205] Tried to move back new op lower bound from 7332736265901199360 to 7332736265868636160. Current Snapshot: MvccSnapshot[applied={T|T < 7332736265901199360}]
I20260924 03:00:14.920789 16206 mvcc.cc:205] Tried to move back new op lower bound from 7332736265901199360 to 7332736265868636160. Current Snapshot: MvccSnapshot[applied={T|T < 7332736265901199360}]
I20260924 03:00:14.922448 15719 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:00:14.922547 15719 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45522:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:00:14.923223 15719 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: Sending DeleteTablet for 3 replicas of tablet d2b24f2afdce4c229cb3eed174ae4328
I20260924 03:00:14.923521 15978 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b24f2afdce4c229cb3eed174ae4328 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:00:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:54700
I20260924 03:00:14.923530 16112 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b24f2afdce4c229cb3eed174ae4328 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:00:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:36316
I20260924 03:00:14.923522 15844 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b24f2afdce4c229cb3eed174ae4328 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:00:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:50106
I20260924 03:00:14.923822 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 15781
I20260924 03:00:14.924441 16216 tablet_replica.cc:330] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39: stopping tablet replica
I20260924 03:00:14.924569 16217 tablet_replica.cc:330] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa: stopping tablet replica
I20260924 03:00:14.924661 16217 raft_consensus.cc:2229] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:00:14.924760 16217 raft_consensus.cc:2258] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:00:14.924923 16216 raft_consensus.cc:2229] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:00:14.925035 16216 raft_consensus.cc:2258] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:00:14.925354 16217 ts_tablet_manager.cc:1905] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:00:14.925431 16216 ts_tablet_manager.cc:1905] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:00:14.927129 16216 ts_tablet_manager.cc:1918] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:00:14.927129 16217 ts_tablet_manager.cc:1918] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:00:14.927198 16216 log.cc:1196] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/d2b24f2afdce4c229cb3eed174ae4328
I20260924 03:00:14.927289 16217 log.cc:1196] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/d2b24f2afdce4c229cb3eed174ae4328
I20260924 03:00:14.927467 16216 ts_tablet_manager.cc:1939] T d2b24f2afdce4c229cb3eed174ae4328 P 69ed3bcd3eea43fbb1cedc5d3776bc39: Deleting consensus metadata
I20260924 03:00:14.927640 16217 ts_tablet_manager.cc:1939] T d2b24f2afdce4c229cb3eed174ae4328 P 237e991937424c92b84adf2579f7a2fa: Deleting consensus metadata
I20260924 03:00:14.927906 15703 catalog_manager.cc:4841] TS 69ed3bcd3eea43fbb1cedc5d3776bc39 (127.0.98.2:41077): tablet d2b24f2afdce4c229cb3eed174ae4328 (table test-table [id=681f8a35551a4bcd9b0499af1bbe82c6]) successfully deleted
I20260924 03:00:14.928081 15704 catalog_manager.cc:4841] TS 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361): tablet d2b24f2afdce4c229cb3eed174ae4328 (table test-table [id=681f8a35551a4bcd9b0499af1bbe82c6]) successfully deleted
W20260924 03:00:14.929865 15706 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38161 (error 108)
I20260924 03:00:14.930053 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 15915
I20260924 03:00:14.933317 16187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:14.933193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38161 (local address 127.0.0.1:50110)
0924 03:00:14.933244 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:14.933284 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260924 03:00:14.933451 15706 catalog_manager.cc:4584] TS 9992277a03de478d9bcccf4c57b0629c (127.0.98.1:38161): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d2b24f2afdce4c229cb3eed174ae4328: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38161: connect: Connection refused (error 111)
I20260924 03:00:14.934971 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 16049
I20260924 03:00:14.939073 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 15687
I20260924 03:00:34.943562 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37721
--webserver_interface=127.0.98.62
--webserver_port=37251
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:00:35.021628 16219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:00:35.021823 16219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:00:35.021844 16219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:00:35.022974 16219 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:00:35.023015 16219 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:00:35.023026 16219 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:00:35.023036 16219 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:00:35.023048 16219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:00:35.023059 16219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:00:35.023070 16219 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:00:35.023082 16219 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:00:35.024482 16219 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:00:35.024667 16219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:00:35.024830 16219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:00:35.032763 16226 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:35.032851 16225 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:35.032745 16228 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:00:35.032938 16219 server_base.cc:1034] running on GCE node
I20260924 03:00:35.033308 16219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:00:35.033613 16219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:00:35.034757 16219 hybrid_clock.cc:648] HybridClock initialized: now 1790218835034737 us; error 35 us; skew 500 ppm
Sep 24 03:00:35 dist-test-slave-sb6p krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:00:35.037444 16219 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:00:35.037993 16219 webserver.cc:460] Webserver started at http://127.0.98.62:37251/ using document root <none> and password file <none>
I20260924 03:00:35.038259 16219 fs_manager.cc:359] Metadata directory not provided
I20260924 03:00:35.038336 16219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:00:35.039832 16219 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:00:35.040431 16235 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:35.040622 16219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:00:35.040709 16219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "0d379a44b74d40919e53fd3d056209e2"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "efe780b253f01f60ae97580cfcfe4151"
server_key_iv: "3a6967d827c111046a13f99ec9fc7a12"
server_key_version: "encryptionkey@0"
I20260924 03:00:35.041155 16219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:00:35.059885 16219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:00:35.061923 16219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:00:35.062044 16219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:00:35.065737 16219 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37721
I20260924 03:00:35.065774 16287 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37721 every 8 connection(s)
I20260924 03:00:35.066111 16219 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:00:35.067620 16288 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:00:35.068266 16288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: Bootstrap starting.
I20260924 03:00:35.068516 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 16219
I20260924 03:00:35.068781 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38161
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37721
--webserver_port=35663
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:00:35.076517 16288 log.cc:824] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:00:35.078295 16288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:00:35.078578 16288 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: Bootstrap complete.
I20260924 03:00:35.080626 16288 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } }
I20260924 03:00:35.080845 16288 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d379a44b74d40919e53fd3d056209e2, State: Initialized, Role: FOLLOWER
I20260924 03:00:35.080994 16288 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } }
I20260924 03:00:35.081070 16288 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:00:35.081123 16288 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:00:35.081167 16288 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 1 FOLLOWER]: Advancing to term 2
I20260924 03:00:35.082168 16288 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } }
I20260924 03:00:35.082260 16288 leader_election.cc:302] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d379a44b74d40919e53fd3d056209e2; no voters:
I20260924 03:00:35.082434 16288 leader_election.cc:288] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260924 03:00:35.082480 16292 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 2 FOLLOWER]: Leader election won for term 2
I20260924 03:00:35.082634 16292 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 2 LEADER]: Becoming Leader. State: Replica: 0d379a44b74d40919e53fd3d056209e2, State: Running, Role: LEADER
I20260924 03:00:35.082701 16288 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:00:35.082751 16292 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } }
I20260924 03:00:35.083030 16292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d379a44b74d40919e53fd3d056209e2. Latest consensus state: current_term: 2 leader_uuid: "0d379a44b74d40919e53fd3d056209e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } } }
I20260924 03:00:35.083057 16293 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "0d379a44b74d40919e53fd3d056209e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } } }
I20260924 03:00:35.083266 16293 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:00:35.083190 16292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:00:35.083506 16301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:00:35.084332 16301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:00:35.084474 16301 catalog_manager.cc:1259] Loaded cluster ID: f9323299bc7348899ed998b3f66b531e
I20260924 03:00:35.084734 16301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:00:35.086000 16301 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:00:35.086246 16301 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: Loaded TSK: 0
I20260924 03:00:35.097371 16301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: Generated new TSK 1
I20260924 03:00:35.097476 16301 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:00:35.149567 16290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:00:35.149777 16290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:00:35.149801 16290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:00:35.150974 16290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:00:35.151013 16290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:00:35.151039 16290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:00:35.152457 16290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35663
--tserver_master_addrs=127.0.98.62:37721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:00:35.152660 16290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:00:35.152871 16290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:00:35.158708 16316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:35.158725 16315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:35.158716 16318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:00:35.158888 16290 server_base.cc:1034] running on GCE node
I20260924 03:00:35.159132 16290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:00:35.159348 16290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:00:35.160495 16290 hybrid_clock.cc:648] HybridClock initialized: now 1790218835160476 us; error 33 us; skew 500 ppm
Sep 24 03:00:35 dist-test-slave-sb6p krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:00:35.163034 16290 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:00:35.163532 16290 webserver.cc:460] Webserver started at http://127.0.98.1:35663/ using document root <none> and password file <none>
I20260924 03:00:35.163736 16290 fs_manager.cc:359] Metadata directory not provided
I20260924 03:00:35.163795 16290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:00:35.165129 16290 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:00:35.165758 16325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:35.165920 16290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:35.166002 16290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "9992277a03de478d9bcccf4c57b0629c"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "9ec74d6137abcdab004bc7f00cb7a8fb"
server_key_iv: "4702dbb084f8b37ba8d54b5f67213d5e"
server_key_version: "encryptionkey@0"
I20260924 03:00:35.166347 16290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:00:35.186323 16290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:00:35.188433 16290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:00:35.188544 16290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:00:35.188769 16290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:00:35.189239 16332 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 03:00:35.190601 16290 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 03:00:35.190651 16290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 03:00:35.190680 16290 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 03:00:35.191282 16290 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 03:00:35.191323 16290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 03:00:35.191382 16332 tablet_bootstrap.cc:492] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c: Bootstrap starting.
I20260924 03:00:35.198689 16290 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38161
I20260924 03:00:35.198745 16439 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38161 every 8 connection(s)
I20260924 03:00:35.199283 16290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:00:35.202569 16332 log.cc:824] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:00:35.203801 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 16290
I20260924 03:00:35.204059 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41077
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37721
--webserver_port=38423
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:00:35.205135 16332 tablet_bootstrap.cc:492] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:00:35.205396 16332 tablet_bootstrap.cc:492] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c: Bootstrap complete.
I20260924 03:00:35.206791 16332 ts_tablet_manager.cc:1397] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.009s
Sep 24 03:00:35 dist-test-slave-sb6p krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:00:35.208231 16332 raft_consensus.cc:348] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:35.208950 16332 raft_consensus.cc:729] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9992277a03de478d9bcccf4c57b0629c, State: Initialized, Role: FOLLOWER
I20260924 03:00:35.209438 16332 consensus_queue.cc:260] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:35.209815 16332 ts_tablet_manager.cc:1428] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260924 03:00:35.212215 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.199982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:60415)
0924 03:00:35.200158 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:35.200202 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:35.200771 (+ 569us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:35.201315 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:35.201319 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:35.201656 (+ 337us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:00:35.201930 (+ 274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:35.201936 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.203618 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.203619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:35.204876 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:35.204879 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.205714 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.205715 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:35.205862 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.206442 (+ 580us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:00:35.206452 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:00:35.208262 (+ 1810us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:00:35.210371 (+ 2109us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:35.210378 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:35.210380 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:35.210694 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:35.211051 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:35.211052 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:35.211053 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:35.211184 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:35.211611 (+ 427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:00:35.211617 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:00:35.211756 (+ 139us) client_negotiation.cc:769] Sending connection context
0924 03:00:35.211834 (+ 78us) client_negotiation.cc:240] Negotiation successful
0924 03:00:35.212013 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":92,"threads_started":1}
I20260924 03:00:35.212745 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.200085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:60415 (local address 127.0.98.62:37721)
0924 03:00:35.200282 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:35.200291 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:35.200336 (+ 45us) server_negotiation.cc:407] Connection header received
0924 03:00:35.201026 (+ 690us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:35.201034 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:35.201116 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:35.201290 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:00:35.202050 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.203488 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.204984 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.205597 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.206159 (+ 562us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.208375 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:00:35.208379 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:00:35.208379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:00:35.208390 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:00:35.210279 (+ 1889us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:35.210781 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:35.210784 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:35.210784 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:35.210884 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:35.211250 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:35.211251 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:35.211251 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:35.211536 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:00:35.212359 (+ 823us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:35.212430 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 03:00:35.212564 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
I20260924 03:00:35.213424 16440 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37721
I20260924 03:00:35.213531 16440 heartbeater.cc:464] Registering TS with master...
I20260924 03:00:35.213708 16440 heartbeater.cc:511] Master 127.0.98.62:37721 requested a full tablet report, sending...
I20260924 03:00:35.214495 16252 ts_manager.cc:194] Registered new tserver with Master: 9992277a03de478d9bcccf4c57b0629c (127.0.98.1:38161)
I20260924 03:00:35.215575 16252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:60415
I20260924 03:00:35.216481 16440 heartbeater.cc:503] Master 127.0.98.62:37721 was elected leader, sending a full tablet report...
Sep 24 03:00:35 dist-test-slave-sb6p krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.1@KRBTEST.COM
I20260924 03:00:35.223601 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.215107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38161 (local address 127.0.0.1:43770)
0924 03:00:35.215479 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:35.215514 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:35.215576 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:35.215917 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:35.215932 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:35.216247 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:00:35.216305 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:35.216307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.217604 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.217605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:35.218221 (+ 616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:35.218224 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.218832 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.218835 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:35.218953 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.219412 (+ 459us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:00:35.219417 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:00:35.220650 (+ 1233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:00:35.222451 (+ 1801us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:35.222456 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:35.222458 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:35.222678 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:35.222868 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:35.222869 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:35.222869 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:35.222937 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:35.223282 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:00:35.223288 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:00:35.223335 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 03:00:35.223428 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 03:00:35.223527 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":83,"threads_started":1}
I20260924 03:00:35.223664 16448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.215339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43770 (local address 127.0.98.1:38161)
0924 03:00:35.215686 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:35.215690 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:35.215699 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:00:35.215729 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:35.215731 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:35.215744 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:35.215798 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:00:35.216389 (+ 591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.217524 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.218293 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.218768 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.218833 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.220728 (+ 1895us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:00:35.220736 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:00:35.220736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:00:35.220743 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:00:35.222373 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:35.222749 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:35.222751 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:35.222752 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:35.222815 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:35.222989 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:35.222989 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:35.222990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:35.223163 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:00:35.223255 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:35.223471 (+ 216us) server_negotiation.cc:299] Negotiation successful
0924 03:00:35.223567 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":61,"threads_started":1}
I20260924 03:00:35.224041 16374 tablet_service.cc:1514] Processing DeleteTablet for tablet d2b24f2afdce4c229cb3eed174ae4328 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:00:14 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43770
I20260924 03:00:35.224519 16449 tablet_replica.cc:330] stopping tablet replica
I20260924 03:00:35.224623 16449 raft_consensus.cc:2229] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:00:35.224684 16449 raft_consensus.cc:2258] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:00:35.224985 16449 ts_tablet_manager.cc:1905] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:00:35.227376 16449 ts_tablet_manager.cc:1918] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:00:35.227449 16449 log.cc:1196] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal/wals/d2b24f2afdce4c229cb3eed174ae4328
I20260924 03:00:35.227648 16449 ts_tablet_manager.cc:1939] T d2b24f2afdce4c229cb3eed174ae4328 P 9992277a03de478d9bcccf4c57b0629c: Deleting consensus metadata
I20260924 03:00:35.228216 16239 catalog_manager.cc:4841] TS 9992277a03de478d9bcccf4c57b0629c (127.0.98.1:38161): tablet d2b24f2afdce4c229cb3eed174ae4328 (table test-table [id=681f8a35551a4bcd9b0499af1bbe82c6]) successfully deleted
W20260924 03:00:35.289072 16445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:00:35.289292 16445 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:00:35.289322 16445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:00:35.290506 16445 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:00:35.290577 16445 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:00:35.290611 16445 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:00:35.292083 16445 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38423
--tserver_master_addrs=127.0.98.62:37721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16445
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:00:35.292315 16445 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:00:35.292531 16445 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:00:35.297335 16457 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:35.297335 16454 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:35.297359 16455 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:00:35.297398 16445 server_base.cc:1034] running on GCE node
I20260924 03:00:35.297765 16445 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:00:35.298009 16445 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:00:35.299150 16445 hybrid_clock.cc:648] HybridClock initialized: now 1790218835299136 us; error 33 us; skew 500 ppm
Sep 24 03:00:35 dist-test-slave-sb6p krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:00:35.301514 16445 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:00:35.301924 16445 webserver.cc:460] Webserver started at http://127.0.98.2:38423/ using document root <none> and password file <none>
I20260924 03:00:35.302127 16445 fs_manager.cc:359] Metadata directory not provided
I20260924 03:00:35.302186 16445 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:00:35.303496 16445 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:00:35.304080 16464 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:35.304241 16445 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:00:35.304319 16445 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "18034413df9f6b211b599017e9c07054"
server_key_iv: "2956a93c75b5c4c412772e391cc96500"
server_key_version: "encryptionkey@0"
I20260924 03:00:35.304638 16445 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:00:35.331815 16445 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:00:35.333562 16445 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:00:35.333707 16445 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:00:35.333936 16445 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:00:35.334226 16445 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:00:35.334259 16445 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:35.334286 16445 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:00:35.334303 16445 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:35.339709 16445 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41077
I20260924 03:00:35.339746 16577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41077 every 8 connection(s)
I20260924 03:00:35.340091 16445 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:00:35 dist-test-slave-sb6p krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:00:35.348639 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.340773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55767 (local address 127.0.98.62:37721)
0924 03:00:35.340816 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:35.340818 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:35.341003 (+ 185us) server_negotiation.cc:407] Connection header received
0924 03:00:35.341510 (+ 507us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:35.341511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:35.341525 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:35.341579 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:00:35.342564 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.343074 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.344132 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.344361 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.344421 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.346387 (+ 1966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:00:35.346388 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:00:35.346389 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:00:35.346396 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:00:35.347516 (+ 1120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:35.347859 (+ 343us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:35.347860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:35.347860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:35.347966 (+ 106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:35.348156 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:35.348157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:35.348157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:35.348261 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:00:35.348306 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:35.348505 (+ 199us) server_negotiation.cc:299] Negotiation successful
0924 03:00:35.348573 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:00:35.348776 16580 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.340752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:55767)
0924 03:00:35.340890 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:35.340925 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:35.341401 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:35.341642 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:35.341645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:35.342242 (+ 597us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:00:35.342433 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:35.342438 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.343146 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.343146 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:35.344043 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:35.344045 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.344428 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.344429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:35.344526 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.345012 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:00:35.345021 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:00:35.346306 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:00:35.347588 (+ 1282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:35.347594 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:35.347596 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:35.347796 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:35.348017 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:35.348018 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:35.348018 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:35.348109 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:35.348301 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:00:35.348303 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:00:35.348410 (+ 107us) client_negotiation.cc:769] Sending connection context
0924 03:00:35.348485 (+ 75us) client_negotiation.cc:240] Negotiation successful
0924 03:00:35.348598 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":75,"threads_started":1}
I20260924 03:00:35.349251 16578 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37721
I20260924 03:00:35.349334 16578 heartbeater.cc:464] Registering TS with master...
I20260924 03:00:35.349498 16578 heartbeater.cc:511] Master 127.0.98.62:37721 requested a full tablet report, sending...
I20260924 03:00:35.349852 16252 ts_manager.cc:194] Registered new tserver with Master: 69ed3bcd3eea43fbb1cedc5d3776bc39 (127.0.98.2:41077)
I20260924 03:00:35.349980 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 16445
I20260924 03:00:35.350234 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35361
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37721
--webserver_port=33441
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:00:35.350270 16252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:55767
W20260924 03:00:35.428391 16581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:00:35.428581 16581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:00:35.428603 16581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:00:35.429716 16581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:00:35.429756 16581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:00:35.429781 16581 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:00:35.431208 16581 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33441
--tserver_master_addrs=127.0.98.62:37721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:00:35.431409 16581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:00:35.431607 16581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:00:35.436379 16589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:35.436389 16587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:35.436398 16586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:00:35.436537 16581 server_base.cc:1034] running on GCE node
I20260924 03:00:35.436739 16581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:00:35.436962 16581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:00:35.438122 16581 hybrid_clock.cc:648] HybridClock initialized: now 1790218835438112 us; error 33 us; skew 500 ppm
Sep 24 03:00:35 dist-test-slave-sb6p krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:00:35.440558 16581 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:00:35.440927 16581 webserver.cc:460] Webserver started at http://127.0.98.3:33441/ using document root <none> and password file <none>
I20260924 03:00:35.441131 16581 fs_manager.cc:359] Metadata directory not provided
I20260924 03:00:35.441185 16581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:00:35.442473 16581 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:00:35.443125 16596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:35.443269 16581 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260924 03:00:35.443346 16581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "237e991937424c92b84adf2579f7a2fa"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "2ea8fa57b029cfc439ff0614a8bd9c1f"
server_key_iv: "8a887b272cb211ef390d0ded6ce2164f"
server_key_version: "encryptionkey@0"
I20260924 03:00:35.443727 16581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:00:35.455444 16581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:00:35.457222 16581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:00:35.457326 16581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:00:35.457491 16581 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:00:35.457718 16581 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:00:35.457746 16581 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:35.457765 16581 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:00:35.457782 16581 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:35.463138 16581 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35361
I20260924 03:00:35.463200 16709 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35361 every 8 connection(s)
I20260924 03:00:35.463495 16581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:00:35.464809 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 16581
Sep 24 03:00:35 dist-test-slave-sb6p krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218835, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:00:35.471547 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.465244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48584 (local address 127.0.98.62:37721)
0924 03:00:35.465339 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:35.465341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:35.465469 (+ 128us) server_negotiation.cc:407] Connection header received
0924 03:00:35.465528 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:35.465528 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:35.465539 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:35.465589 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:00:35.466241 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.466787 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.467508 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.467703 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.467757 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.468999 (+ 1242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:00:35.469001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:00:35.469001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:00:35.469008 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:00:35.470386 (+ 1378us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:35.470749 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:35.470750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:35.470751 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:35.470803 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:35.471027 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:35.471028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:35.471028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:35.471137 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:00:35.471318 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:35.471352 (+ 34us) server_negotiation.cc:299] Negotiation successful
0924 03:00:35.471445 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79,"thread_start_us":48,"threads_started":1}
I20260924 03:00:35.472640 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.464165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:40189 (local address 127.0.98.62:37721)
0924 03:00:35.464244 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:35.464247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:35.464455 (+ 208us) server_negotiation.cc:407] Connection header received
0924 03:00:35.464915 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:35.464915 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:35.464927 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:35.464994 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:00:35.465973 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.466610 (+ 637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.467582 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.467774 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.467826 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.470056 (+ 2230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:00:35.470057 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:00:35.470058 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:00:35.470064 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:00:35.471340 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:35.471788 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:35.471789 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:35.471789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:35.471845 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:35.472161 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:35.472162 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:35.472162 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:35.472257 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:00:35.472308 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:35.472530 (+ 222us) server_negotiation.cc:299] Negotiation successful
0924 03:00:35.472585 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 03:00:35.472836 16712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.464155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:40189)
0924 03:00:35.464320 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:35.464354 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:35.464818 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:35.465054 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:35.465058 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:35.465696 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:00:35.465886 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:35.465891 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.466696 (+ 805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.466697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:35.467501 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:35.467504 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.467839 (+ 335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.467840 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:35.467983 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.468413 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:00:35.468422 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:00:35.469976 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:00:35.471410 (+ 1434us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:35.471416 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:35.471418 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:35.471696 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:35.471914 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:35.471914 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:35.471915 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:35.472042 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:35.472311 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:00:35.472315 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:00:35.472430 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 03:00:35.472510 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 03:00:35.472625 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":96,"threads_started":1}
I20260924 03:00:35.473271 16710 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37721
I20260924 03:00:35.473343 16710 heartbeater.cc:464] Registering TS with master...
I20260924 03:00:35.473495 16710 heartbeater.cc:511] Master 127.0.98.62:37721 requested a full tablet report, sending...
I20260924 03:00:35.473788 16252 ts_manager.cc:194] Registered new tserver with Master: 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361)
I20260924 03:00:35.474128 16252 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:40189
I20260924 03:00:35.483013 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:00:35.485644 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.483370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48598 (local address 127.0.98.62:37721)
0924 03:00:35.483412 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:35.483414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:35.483544 (+ 130us) server_negotiation.cc:407] Connection header received
0924 03:00:35.483618 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:35.483618 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:35.483651 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:35.483702 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:00:35.483781 (+ 79us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.484272 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.485032 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.485209 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.485258 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.485435 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:00:35.485476 (+ 41us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0924 03:00:35.485594 (+ 118us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48598: token expired
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 03:00:35.485692 16443 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48598: token expired
W20260924 03:00:35.485803 16181 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.98.62:37721)
I20260924 03:00:35.491477 16443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.486081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48610 (local address 127.0.98.62:37721)
0924 03:00:35.486135 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:35.486137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:35.486144 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:00:35.486165 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:35.486166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:35.486173 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:35.486215 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:00:35.486782 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.487300 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.487984 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.488179 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.488237 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.489294 (+ 1057us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:00:35.489294 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:00:35.489295 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:00:35.489301 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:00:35.490664 (+ 1363us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:35.490948 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:35.490949 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:35.490949 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:35.490998 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:35.491141 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:35.491141 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:35.491142 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:35.491230 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:00:35.491297 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:35.491362 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 03:00:35.491412 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260924 03:00:35.492533 16252 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:00:35.494078 16252 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:00:35.496083 16374 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed32090402f40e19b46d6fdbee8ee9d (DEFAULT_TABLE table=test-table [id=740d34be3d1447319aa13ab11e469e45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:00:35.496279 16374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed32090402f40e19b46d6fdbee8ee9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:00:35.497650 16332 tablet_bootstrap.cc:492] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c: Bootstrap starting.
I20260924 03:00:35.497977 16332 tablet_bootstrap.cc:654] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c: Neither blocks nor log segments found. Creating new log.
I20260924 03:00:35.498453 16332 tablet_bootstrap.cc:492] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c: No bootstrap required, opened a new log
I20260924 03:00:35.498495 16332 ts_tablet_manager.cc:1397] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.000s
I20260924 03:00:35.498862 16332 raft_consensus.cc:348] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:35.498934 16332 raft_consensus.cc:374] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:00:35.498971 16332 raft_consensus.cc:729] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9992277a03de478d9bcccf4c57b0629c, State: Initialized, Role: FOLLOWER
I20260924 03:00:35.499032 16332 consensus_queue.cc:260] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:35.499118 16332 ts_tablet_manager.cc:1428] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260924 03:00:35.499701 16716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.495849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41077 (local address 127.0.0.1:60600)
0924 03:00:35.496073 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:35.496081 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:35.496114 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:35.496432 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:35.496433 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:35.496436 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:00:35.496479 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:35.496481 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.497414 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.497415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:35.498550 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:35.498553 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.499459 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.499460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:35.499521 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.499541 (+ 20us) client_negotiation.cc:769] Sending connection context
0924 03:00:35.499588 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 03:00:35.499628 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":100,"threads_started":1}
I20260924 03:00:35.499907 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.496136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60600 (local address 127.0.98.2:41077)
0924 03:00:35.496281 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:35.496286 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:35.496294 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:00:35.496331 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:35.496332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:35.496344 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:35.496415 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:00:35.496547 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.497333 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.498633 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.499380 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.499449 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.499534 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:35.499796 (+ 262us) server_negotiation.cc:299] Negotiation successful
0924 03:00:35.499834 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":78,"threads_started":1}
I20260924 03:00:35.500551 16512 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed32090402f40e19b46d6fdbee8ee9d (DEFAULT_TABLE table=test-table [id=740d34be3d1447319aa13ab11e469e45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:00:35.500845 16512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed32090402f40e19b46d6fdbee8ee9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:00:35.500890 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.495828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35361 (local address 127.0.0.1:59318)
0924 03:00:35.495944 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:35.495956 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:35.495991 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:35.497872 (+ 1881us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:35.497872 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:35.497874 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:00:35.497912 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:35.497913 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.499047 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.499048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:35.499972 (+ 924us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:35.499975 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.500669 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.500669 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:35.500738 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.500742 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:00:35.500787 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 03:00:35.500827 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100}
I20260924 03:00:35.501070 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.495905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59318 (local address 127.0.98.3:35361)
0924 03:00:35.497715 (+ 1810us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:35.497719 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:35.497730 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:00:35.497766 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:35.497768 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:35.497784 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:35.497867 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:00:35.497976 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.498974 (+ 998us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.500070 (+ 1096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.500592 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.500654 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.500711 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:35.500797 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 03:00:35.500833 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1773,"thread_start_us":91,"threads_started":1}
I20260924 03:00:35.501648 16644 tablet_service.cc:1467] Processing CreateTablet for tablet 4ed32090402f40e19b46d6fdbee8ee9d (DEFAULT_TABLE table=test-table [id=740d34be3d1447319aa13ab11e469e45]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:00:35.501906 16644 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ed32090402f40e19b46d6fdbee8ee9d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:00:35.503564 16720 tablet_bootstrap.cc:492] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39: Bootstrap starting.
I20260924 03:00:35.504251 16720 tablet_bootstrap.cc:654] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39: Neither blocks nor log segments found. Creating new log.
I20260924 03:00:35.504566 16721 tablet_bootstrap.cc:492] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa: Bootstrap starting.
I20260924 03:00:35.504805 16720 log.cc:824] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39: Log is configured to *not* fsync() on all Append() calls
I20260924 03:00:35.505087 16721 tablet_bootstrap.cc:654] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa: Neither blocks nor log segments found. Creating new log.
I20260924 03:00:35.505417 16720 tablet_bootstrap.cc:492] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39: No bootstrap required, opened a new log
I20260924 03:00:35.505468 16720 ts_tablet_manager.cc:1397] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:00:35.505563 16721 log.cc:824] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa: Log is configured to *not* fsync() on all Append() calls
I20260924 03:00:35.506179 16721 tablet_bootstrap.cc:492] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa: No bootstrap required, opened a new log
I20260924 03:00:35.506235 16721 ts_tablet_manager.cc:1397] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:00:35.506991 16720 raft_consensus.cc:348] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:35.507130 16720 raft_consensus.cc:374] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:00:35.507207 16720 raft_consensus.cc:729] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69ed3bcd3eea43fbb1cedc5d3776bc39, State: Initialized, Role: FOLLOWER
I20260924 03:00:35.507344 16720 consensus_queue.cc:260] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:35.507622 16720 ts_tablet_manager.cc:1428] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:00:35.507733 16578 heartbeater.cc:503] Master 127.0.98.62:37721 was elected leader, sending a full tablet report...
I20260924 03:00:35.507834 16721 raft_consensus.cc:348] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:35.508001 16721 raft_consensus.cc:374] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:00:35.508038 16721 raft_consensus.cc:729] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 237e991937424c92b84adf2579f7a2fa, State: Initialized, Role: FOLLOWER
I20260924 03:00:35.508176 16721 consensus_queue.cc:260] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:35.508442 16721 ts_tablet_manager.cc:1428] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:00:35.508455 16710 heartbeater.cc:503] Master 127.0.98.62:37721 was elected leader, sending a full tablet report...
I20260924 03:00:35.589541 16446 raft_consensus.cc:482] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:00:35.589684 16446 raft_consensus.cc:504] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:35.590077 16446 leader_election.cc:288] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361), 69ed3bcd3eea43fbb1cedc5d3776bc39 (127.0.98.2:41077)
W20260924 03:00:35.590631 16579 tablet.cc:2369] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:00:35.593009 16718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.590321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42159 (local address 127.0.98.2:41077)
0924 03:00:35.590365 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:35.590367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:35.590386 (+ 19us) server_negotiation.cc:407] Connection header received
0924 03:00:35.590471 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:35.590473 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:35.590485 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:35.590579 (+ 94us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:00:35.590757 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.591262 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.592219 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.592696 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.592746 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.592754 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:35.592925 (+ 171us) server_negotiation.cc:299] Negotiation successful
0924 03:00:35.592955 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:00:35.593041 16442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.590242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41077 (local address 127.0.98.1:42159)
0924 03:00:35.590296 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:35.590310 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:35.590381 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:35.590586 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:35.590588 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:35.590590 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:00:35.590658 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:35.590661 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.591336 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.591337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:35.592137 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:35.592140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.592759 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.592760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:35.592834 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.592858 (+ 24us) client_negotiation.cc:769] Sending connection context
0924 03:00:35.592916 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 03:00:35.592956 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 03:00:35.593340 16717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.590352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:49335 (local address 127.0.98.3:35361)
0924 03:00:35.590397 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:35.590399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:35.590559 (+ 160us) server_negotiation.cc:407] Connection header received
0924 03:00:35.590677 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:35.590678 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:35.590691 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:35.590739 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:00:35.590937 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.591463 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.592517 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.592966 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.593022 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.593031 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:35.593239 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 03:00:35.593282 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:00:35.593345 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.590251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35361 (local address 127.0.98.1:49335)
0924 03:00:35.590463 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:35.590472 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:35.590586 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:35.590744 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:35.590745 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:35.590747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:00:35.590814 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:35.590815 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.591544 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.591545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:35.592439 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:35.592442 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.593042 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.593043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:35.593127 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.593132 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:00:35.593212 (+ 80us) client_negotiation.cc:240] Negotiation successful
0924 03:00:35.593247 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":54,"thread_start_us":103,"threads_started":1}
I20260924 03:00:35.593357 16532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed32090402f40e19b46d6fdbee8ee9d" candidate_uuid: "9992277a03de478d9bcccf4c57b0629c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" is_pre_election: true
I20260924 03:00:35.593497 16532 raft_consensus.cc:2454] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9992277a03de478d9bcccf4c57b0629c in term 0.
I20260924 03:00:35.593676 16664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed32090402f40e19b46d6fdbee8ee9d" candidate_uuid: "9992277a03de478d9bcccf4c57b0629c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "237e991937424c92b84adf2579f7a2fa" is_pre_election: true
I20260924 03:00:35.593708 16326 leader_election.cc:302] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ed3bcd3eea43fbb1cedc5d3776bc39, 9992277a03de478d9bcccf4c57b0629c; no voters:
I20260924 03:00:35.593804 16664 raft_consensus.cc:2454] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9992277a03de478d9bcccf4c57b0629c in term 0.
I20260924 03:00:35.593830 16446 raft_consensus.cc:2781] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:00:35.593895 16446 raft_consensus.cc:482] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:00:35.593924 16446 raft_consensus.cc:3037] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:00:35.594655 16446 raft_consensus.cc:504] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:35.594763 16446 leader_election.cc:288] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 1 election: Requested vote from peers 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361), 69ed3bcd3eea43fbb1cedc5d3776bc39 (127.0.98.2:41077)
I20260924 03:00:35.594920 16664 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed32090402f40e19b46d6fdbee8ee9d" candidate_uuid: "9992277a03de478d9bcccf4c57b0629c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "237e991937424c92b84adf2579f7a2fa"
I20260924 03:00:35.594941 16532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ed32090402f40e19b46d6fdbee8ee9d" candidate_uuid: "9992277a03de478d9bcccf4c57b0629c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39"
I20260924 03:00:35.594990 16664 raft_consensus.cc:3037] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:00:35.595002 16532 raft_consensus.cc:3037] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:00:35.595647 16664 raft_consensus.cc:2454] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9992277a03de478d9bcccf4c57b0629c in term 1.
I20260924 03:00:35.595662 16532 raft_consensus.cc:2454] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9992277a03de478d9bcccf4c57b0629c in term 1.
I20260924 03:00:35.595795 16326 leader_election.cc:302] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 69ed3bcd3eea43fbb1cedc5d3776bc39, 9992277a03de478d9bcccf4c57b0629c; no voters:
I20260924 03:00:35.595871 16446 raft_consensus.cc:2781] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:00:35.595989 16446 raft_consensus.cc:686] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 LEADER]: Becoming Leader. State: Replica: 9992277a03de478d9bcccf4c57b0629c, State: Running, Role: LEADER
I20260924 03:00:35.596045 16446 consensus_queue.cc:237] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:35.596736 16252 catalog_manager.cc:5495] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c reported cstate change: term changed from 0 to 1, leader changed from <none> to 9992277a03de478d9bcccf4c57b0629c (127.0.98.1). New cstate: current_term: 1 leader_uuid: "9992277a03de478d9bcccf4c57b0629c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } health_report { overall_health: HEALTHY } } }
I20260924 03:00:35.614288 16448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.612273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43776 (local address 127.0.98.1:38161)
0924 03:00:35.612317 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:35.612319 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:35.612348 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:00:35.612380 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:35.612380 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:35.612415 (+ 35us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:35.612468 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:00:35.612564 (+ 96us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.613049 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.613712 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:35.613908 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:35.613962 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:35.614051 (+ 89us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:00:35.614144 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:00:35.614186 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:35.614213 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 03:00:35.614231 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:00:35.615734 16664 raft_consensus.cc:1261] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa [term 1 FOLLOWER]: Refusing update from remote peer 9992277a03de478d9bcccf4c57b0629c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:00:35.615721 16532 raft_consensus.cc:1261] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 1 FOLLOWER]: Refusing update from remote peer 9992277a03de478d9bcccf4c57b0629c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:00:35.616120 16727 consensus_queue.cc:1035] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [LEADER]: Connected to new peer: Peer: permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:00:35.616267 16446 consensus_queue.cc:1035] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [LEADER]: Connected to new peer: Peer: permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:00:35.618186 16730 mvcc.cc:205] Tried to move back new op lower bound from 7332736350679642112 to 7332736350601793536. Current Snapshot: MvccSnapshot[applied={T|T < 7332736350679642112}]
I20260924 03:00:35.621023 16252 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:00:35.621114 16252 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:00:35.621806 16252 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: Sending DeleteTablet for 3 replicas of tablet 4ed32090402f40e19b46d6fdbee8ee9d
I20260924 03:00:35.622108 16374 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ed32090402f40e19b46d6fdbee8ee9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:00:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:43770
I20260924 03:00:35.622126 16512 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ed32090402f40e19b46d6fdbee8ee9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:00:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:60600
I20260924 03:00:35.622193 16449 tablet_replica.cc:330] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c: stopping tablet replica
I20260924 03:00:35.622186 16644 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ed32090402f40e19b46d6fdbee8ee9d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:00:35 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:59318
I20260924 03:00:35.622234 16449 raft_consensus.cc:2229] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:00:35.622314 16449 raft_consensus.cc:2258] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:00:35.622385 16741 tablet_replica.cc:330] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa: stopping tablet replica
I20260924 03:00:35.622483 16740 tablet_replica.cc:330] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39: stopping tablet replica
I20260924 03:00:35.622491 16741 raft_consensus.cc:2229] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:00:35.622493 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 16290
I20260924 03:00:35.622593 16740 raft_consensus.cc:2229] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:00:35.622606 16741 raft_consensus.cc:2258] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:00:35.622680 16740 raft_consensus.cc:2258] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:00:35.623239 16741 ts_tablet_manager.cc:1905] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:00:35.623411 16740 ts_tablet_manager.cc:1905] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:00:35.624698 16741 ts_tablet_manager.cc:1918] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:00:35.624744 16741 log.cc:1196] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/4ed32090402f40e19b46d6fdbee8ee9d
I20260924 03:00:35.624949 16741 ts_tablet_manager.cc:1939] T 4ed32090402f40e19b46d6fdbee8ee9d P 237e991937424c92b84adf2579f7a2fa: Deleting consensus metadata
I20260924 03:00:35.624981 16740 ts_tablet_manager.cc:1918] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:00:35.625039 16740 log.cc:1196] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/4ed32090402f40e19b46d6fdbee8ee9d
I20260924 03:00:35.625275 16740 ts_tablet_manager.cc:1939] T 4ed32090402f40e19b46d6fdbee8ee9d P 69ed3bcd3eea43fbb1cedc5d3776bc39: Deleting consensus metadata
I20260924 03:00:35.625269 16237 catalog_manager.cc:4841] TS 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361): tablet 4ed32090402f40e19b46d6fdbee8ee9d (table test-table [id=740d34be3d1447319aa13ab11e469e45]) successfully deleted
I20260924 03:00:35.625707 16236 catalog_manager.cc:4841] TS 69ed3bcd3eea43fbb1cedc5d3776bc39 (127.0.98.2:41077): tablet 4ed32090402f40e19b46d6fdbee8ee9d (table test-table [id=740d34be3d1447319aa13ab11e469e45]) successfully deleted
W20260924 03:00:35.629352 16239 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:38161 (error 108)
I20260924 03:00:35.629524 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 16445
I20260924 03:00:35.630055 16447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:35.629678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:38161 (local address 127.0.0.1:43784)
0924 03:00:35.629984 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:35.630023 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38161: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":286}
W20260924 03:00:35.630162 16239 catalog_manager.cc:4584] TS 9992277a03de478d9bcccf4c57b0629c (127.0.98.1:38161): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4ed32090402f40e19b46d6fdbee8ee9d: Network error: Client connection negotiation failed: client connection to 127.0.98.1:38161: connect: Connection refused (error 111)
I20260924 03:00:35.634181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 16581
I20260924 03:00:35.638549 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 16219
I20260924 03:00:55.643000 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37721
--webserver_interface=127.0.98.62
--webserver_port=37251
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37721
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:00:55.721920 16743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:00:55.722122 16743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:00:55.722151 16743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:00:55.723295 16743 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260924 03:00:55.723335 16743 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260924 03:00:55.723351 16743 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260924 03:00:55.723366 16743 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:00:55.723380 16743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:00:55.723399 16743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:00:55.723419 16743 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:00:55.723443 16743 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:00:55.724900 16743 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37721
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37721
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=37251
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16743
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:00:55.725142 16743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:00:55.725342 16743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:00:55.730347 16743 server_base.cc:1034] running on GCE node
W20260924 03:00:55.730368 16749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:55.730410 16751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:55.730494 16748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:00:55.730826 16743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:00:55.731076 16743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:00:55.732244 16743 hybrid_clock.cc:648] HybridClock initialized: now 1790218855732224 us; error 32 us; skew 500 ppm
Sep 24 03:00:55 dist-test-slave-sb6p krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:00:55.734740 16743 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:00:55.735149 16743 webserver.cc:460] Webserver started at http://127.0.98.62:37251/ using document root <none> and password file <none>
I20260924 03:00:55.735355 16743 fs_manager.cc:359] Metadata directory not provided
I20260924 03:00:55.735404 16743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:00:55.736693 16743 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:00:55.737360 16758 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:55.737512 16743 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:00:55.737634 16743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "0d379a44b74d40919e53fd3d056209e2"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "efe780b253f01f60ae97580cfcfe4151"
server_key_iv: "3a6967d827c111046a13f99ec9fc7a12"
server_key_version: "encryptionkey@0"
I20260924 03:00:55.737988 16743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:00:55.745816 16743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:00:55.747622 16743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:00:55.747751 16743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:00:55.751567 16743 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37721
I20260924 03:00:55.751607 16810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37721 every 8 connection(s)
I20260924 03:00:55.751950 16743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:00:55.753324 16811 sys_catalog.cc:263] Verifying existing consensus state
I20260924 03:00:55.753901 16811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: Bootstrap starting.
I20260924 03:00:55.757555 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 16743
I20260924 03:00:55.757890 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:38161
--local_ip_for_outbound_sockets=127.0.98.1
--tserver_master_addrs=127.0.98.62:37721
--webserver_port=35663
--webserver_interface=127.0.98.1
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:00:55.762295 16811 log.cc:824] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:00:55.764494 16811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:00:55.764753 16811 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: Bootstrap complete.
I20260924 03:00:55.766775 16811 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } }
I20260924 03:00:55.766989 16811 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d379a44b74d40919e53fd3d056209e2, State: Initialized, Role: FOLLOWER
I20260924 03:00:55.767129 16811 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } }
I20260924 03:00:55.767185 16811 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:00:55.767234 16811 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:00:55.767280 16811 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 2 FOLLOWER]: Advancing to term 3
I20260924 03:00:55.768208 16811 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } }
I20260924 03:00:55.768301 16811 leader_election.cc:302] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d379a44b74d40919e53fd3d056209e2; no voters:
I20260924 03:00:55.768450 16811 leader_election.cc:288] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260924 03:00:55.768493 16815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 3 FOLLOWER]: Leader election won for term 3
I20260924 03:00:55.768651 16815 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [term 3 LEADER]: Becoming Leader. State: Replica: 0d379a44b74d40919e53fd3d056209e2, State: Running, Role: LEADER
I20260924 03:00:55.768714 16811 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:00:55.769863 16815 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } }
I20260924 03:00:55.770352 16816 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d379a44b74d40919e53fd3d056209e2. Latest consensus state: current_term: 3 leader_uuid: "0d379a44b74d40919e53fd3d056209e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } } }
I20260924 03:00:55.770510 16816 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:00:55.770489 16817 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "0d379a44b74d40919e53fd3d056209e2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d379a44b74d40919e53fd3d056209e2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37721 } } }
I20260924 03:00:55.770794 16817 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:00:55.770918 16829 catalog_manager.cc:1259] Loaded cluster ID: f9323299bc7348899ed998b3f66b531e
I20260924 03:00:55.770962 16829 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: loading cluster ID for follower catalog manager: success
I20260924 03:00:55.771802 16829 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d379a44b74d40919e53fd3d056209e2: acquiring CA information for follower catalog manager: success
F20260924 03:00:55.772105 16829 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fee22d729ad google::LogMessage::Fail() at ??:0
@ 0x7fee22d7588a google::LogMessage::SendToLog() at ??:0
@ 0x7fee22d724de google::LogMessage::Flush() at ??:0
@ 0x7fee22d73a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fee254df104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fee2accb6b9 kudu::master::CatalogManager::LoadTspkEntries() at ??:0
@ 0x7fee2accb989 kudu::master::CatalogManager::PrepareFollowerTokenVerifier() at ??:0
@ 0x7fee2accbe4e kudu::master::CatalogManager::PrepareFollower() at ??:0
@ 0x7fee2acce8c2 kudu::master::CatalogManagerBgTasks::Run() at ??:0
@ 0x7fee24c35954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fee25a996db start_thread at ??:0
@ 0x7fee2187061f clone at ??:0
W20260924 03:00:55.854425 16813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:00:55.854717 16813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:00:55.854748 16813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:00:55.856326 16813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:00:55.856365 16813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:00:55.856400 16813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:00:55.858381 16813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:38161
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=35663
--tserver_master_addrs=127.0.98.62:37721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:00:55.858649 16813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:00:55.858899 16813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:00:55.864971 16849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:00:55.865057 16813 server_base.cc:1034] running on GCE node
W20260924 03:00:55.864995 16852 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:55.865128 16850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:00:55.865428 16813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:00:55.865659 16813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:00:55.866873 16813 hybrid_clock.cc:648] HybridClock initialized: now 1790218855866859 us; error 25 us; skew 500 ppm
Sep 24 03:00:55 dist-test-slave-sb6p krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:00:55.869546 16813 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:00:55.870018 16813 webserver.cc:460] Webserver started at http://127.0.98.1:35663/ using document root <none> and password file <none>
I20260924 03:00:55.870272 16813 fs_manager.cc:359] Metadata directory not provided
I20260924 03:00:55.870322 16813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:00:55.871616 16813 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:00:55.872223 16859 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:55.872412 16813 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:00:55.872511 16813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "9992277a03de478d9bcccf4c57b0629c"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "9ec74d6137abcdab004bc7f00cb7a8fb"
server_key_iv: "4702dbb084f8b37ba8d54b5f67213d5e"
server_key_version: "encryptionkey@0"
I20260924 03:00:55.872956 16813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:00:55.893146 16813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:00:55.895742 16813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:00:55.895885 16813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:00:55.896131 16813 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:00:55.896637 16866 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260924 03:00:55.898044 16813 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260924 03:00:55.898106 16813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260924 03:00:55.898141 16813 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260924 03:00:55.899024 16813 ts_tablet_manager.cc:610] Registered 1 tablets
I20260924 03:00:55.899068 16813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260924 03:00:55.899133 16866 tablet_bootstrap.cc:492] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c: Bootstrap starting.
I20260924 03:00:55.905987 16813 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:38161
I20260924 03:00:55.906143 16973 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:38161 every 8 connection(s)
I20260924 03:00:55.906432 16813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:00:55.908591 16866 log.cc:824] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:00:55.911260 16866 tablet_bootstrap.cc:492] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260924 03:00:55.911542 16866 tablet_bootstrap.cc:492] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c: Bootstrap complete.
I20260924 03:00:55.912613 16866 ts_tablet_manager.cc:1397] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260924 03:00:55.913187 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 16813
I20260924 03:00:55.913414 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:41077
--local_ip_for_outbound_sockets=127.0.98.2
--tserver_master_addrs=127.0.98.62:37721
--webserver_port=38423
--webserver_interface=127.0.98.2
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:00:55.913998 16866 raft_consensus.cc:348] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:55.914325 16866 raft_consensus.cc:729] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9992277a03de478d9bcccf4c57b0629c, State: Initialized, Role: FOLLOWER
I20260924 03:00:55.914467 16866 consensus_queue.cc:260] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:55.914765 16866 ts_tablet_manager.cc:1428] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
W20260924 03:00:55.957715 16976 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:00:55.907284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:41843)
0924 03:00:55.907477 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:55.907527 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:55.908188 (+ 661us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:55.957511 (+ 49323us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":95,"threads_started":1}
W20260924 03:00:55.957995 16974 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37721 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37721: Client connection negotiation failed: client connection to 127.0.98.62:37721: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260924 03:00:55.958330 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:55.958218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:34027)
0924 03:00:55.958262 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:55.958306 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260924 03:00:55.958637 16976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:55.958572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:37861)
0924 03:00:55.958597 (+ 25us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:55.958620 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 03:00:55.958735 16974 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260924 03:00:55.992604 16978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:00:55.992816 16978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:00:55.992841 16978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:00:55.993973 16978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:00:55.994009 16978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:00:55.994033 16978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:00:55.995537 16978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:41077
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=38423
--tserver_master_addrs=127.0.98.62:37721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16978
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:00:55.995725 16978 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:00:55.995903 16978 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:00:56.000659 16984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:56.000694 16985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:56.000695 16987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:00:56.000741 16978 server_base.cc:1034] running on GCE node
I20260924 03:00:56.001072 16978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:00:56.001302 16978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:00:56.002449 16978 hybrid_clock.cc:648] HybridClock initialized: now 1790218856002433 us; error 27 us; skew 500 ppm
Sep 24 03:00:55 dist-test-slave-sb6p krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:00:56.004856 16978 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:00:56.005370 16978 webserver.cc:460] Webserver started at http://127.0.98.2:38423/ using document root <none> and password file <none>
I20260924 03:00:56.005587 16978 fs_manager.cc:359] Metadata directory not provided
I20260924 03:00:56.005635 16978 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:00:56.006942 16978 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:00:56.007513 16994 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:56.007644 16978 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:00:56.007717 16978 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "18034413df9f6b211b599017e9c07054"
server_key_iv: "2956a93c75b5c4c412772e391cc96500"
server_key_version: "encryptionkey@0"
I20260924 03:00:56.008042 16978 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:00:56.026655 16978 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:00:56.028972 16978 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:00:56.029088 16978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:00:56.029297 16978 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:00:56.029546 16978 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:00:56.029574 16978 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:56.029596 16978 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:00:56.029608 16978 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:56.035313 16978 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41077
I20260924 03:00:56.035367 17107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41077 every 8 connection(s)
I20260924 03:00:56.035683 16978 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:00:56.036690 17110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:56.036333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:39793)
0924 03:00:56.036473 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:56.036547 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
W20260924 03:00:56.037034 17108 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37721 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37721: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
I20260924 03:00:56.037277 17110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:56.037187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:34109)
0924 03:00:56.037233 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:56.037260 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 03:00:56.037498 17110 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:56.037439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:39873)
0924 03:00:56.037469 (+ 30us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:56.037487 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 03:00:56.037577 17108 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:00:56.037969 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 16978
I20260924 03:00:56.038198 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:35361
--local_ip_for_outbound_sockets=127.0.98.3
--tserver_master_addrs=127.0.98.62:37721
--webserver_port=33441
--webserver_interface=127.0.98.3
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:37265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:00:56.117478 17111 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:00:56.117677 17111 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:00:56.117709 17111 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:00:56.118872 17111 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:00:56.118912 17111 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:00:56.118947 17111 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:00:56.120370 17111 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:37265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:35361
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=33441
--tserver_master_addrs=127.0.98.62:37721
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17111
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:00:56.120595 17111 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:00:56.120810 17111 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:00:56.125892 17116 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:56.125887 17119 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:00:56.125902 17117 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:00:56.126137 17111 server_base.cc:1034] running on GCE node
I20260924 03:00:56.126380 17111 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:00:56.126605 17111 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:00:56.127766 17111 hybrid_clock.cc:648] HybridClock initialized: now 1790218856127741 us; error 33 us; skew 500 ppm
Sep 24 03:00:56 dist-test-slave-sb6p krb5kdc[15674](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:00:56.130331 17111 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:00:56.130757 17111 webserver.cc:460] Webserver started at http://127.0.98.3:33441/ using document root <none> and password file <none>
I20260924 03:00:56.130949 17111 fs_manager.cc:359] Metadata directory not provided
I20260924 03:00:56.130991 17111 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:00:56.132254 17111 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:00:56.132835 17126 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:56.132962 17111 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:00:56.133044 17111 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "237e991937424c92b84adf2579f7a2fa"
format_stamp: "Formatted at 2026-09-24 03:00:14 on dist-test-slave-sb6p"
server_key: "2ea8fa57b029cfc439ff0614a8bd9c1f"
server_key_iv: "8a887b272cb211ef390d0ded6ce2164f"
server_key_version: "encryptionkey@0"
I20260924 03:00:56.133358 17111 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:00:56.153080 17111 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:00:56.154906 17111 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:00:56.155043 17111 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:00:56.155263 17111 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:00:56.155556 17111 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:00:56.155598 17111 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:56.155630 17111 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:00:56.155653 17111 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:00:56.161072 17111 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35361
I20260924 03:00:56.161142 17239 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35361 every 8 connection(s)
I20260924 03:00:56.161454 17111 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:00:56.162421 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:56.162066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:36841)
0924 03:00:56.162212 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:56.162284 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":76,"threads_started":1}
I20260924 03:00:56.162673 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 17111
W20260924 03:00:56.162907 17240 heartbeater.cc:644] Failed to heartbeat to 127.0.98.62:37721 (0 consecutive failures): Network error: Failed to ping master at 127.0.98.62:37721: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
I20260924 03:00:56.163257 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:56.163108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:34409)
0924 03:00:56.163145 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:56.163229 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260924 03:00:56.163546 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:56.163480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:34283)
0924 03:00:56.163512 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:56.163532 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 03:00:56.163655 17240 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260924 03:00:56.959333 17244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:56.959091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:55955)
0924 03:00:56.959222 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:56.959266 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":58,"threads_started":1}
I20260924 03:00:57.038147 17245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:57.037942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:43089)
0924 03:00:57.038058 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:57.038096 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":58,"threads_started":1}
I20260924 03:00:57.122126 17246 raft_consensus.cc:482] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:00:57.122258 17246 raft_consensus.cc:504] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:57.122573 17246 leader_election.cc:288] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361), 69ed3bcd3eea43fbb1cedc5d3776bc39 (127.0.98.2:41077)
Sep 24 03:00:57 dist-test-slave-sb6p krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.3@KRBTEST.COM
Sep 24 03:00:57 dist-test-slave-sb6p krb5kdc[15674](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.2@KRBTEST.COM
I20260924 03:00:57.132383 17249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:57.122846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55295 (local address 127.0.98.3:35361)
0924 03:00:57.123038 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:57.123045 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:57.123062 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:00:57.123465 (+ 403us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:57.123467 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:57.123519 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:57.123653 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:00:57.124448 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:57.125297 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:57.126283 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:57.126712 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:57.126805 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:57.128990 (+ 2185us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:00:57.128993 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:00:57.128993 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:00:57.129003 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:00:57.130766 (+ 1763us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:57.131258 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:57.131262 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:57.131262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:57.131367 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:57.131600 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:57.131601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:57.131602 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:57.131830 (+ 228us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:00:57.131899 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:57.132172 (+ 273us) server_negotiation.cc:299] Negotiation successful
0924 03:00:57.132285 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":48,"threads_started":1}
I20260924 03:00:57.132387 17248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:57.122816 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:33479 (local address 127.0.98.2:41077)
0924 03:00:57.122998 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:00:57.123010 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0924 03:00:57.123046 (+ 36us) server_negotiation.cc:407] Connection header received
0924 03:00:57.123491 (+ 445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:00:57.123493 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:00:57.123532 (+ 39us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:00:57.123655 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:00:57.124438 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:57.125281 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:57.126313 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:00:57.126719 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:00:57.126819 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:57.129392 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:00:57.129396 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:00:57.129397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:00:57.129406 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:00:57.130970 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:57.131346 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:57.131348 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:57.131348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:57.131440 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:00:57.131603 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:00:57.131604 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:00:57.131605 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:00:57.131828 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:00:57.131895 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 03:00:57.132166 (+ 271us) server_negotiation.cc:299] Negotiation successful
0924 03:00:57.132279 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144,"thread_start_us":61,"threads_started":1}
I20260924 03:00:57.132412 17244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:57.122720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41077 (local address 127.0.98.1:33479)
0924 03:00:57.122840 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:57.122851 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:57.122927 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:57.123707 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:57.123710 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:57.124112 (+ 402us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:00:57.124316 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:57.124322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:57.125371 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:57.125372 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:57.126194 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:57.126196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:57.126792 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:57.126793 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:57.126907 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:57.127429 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:00:57.127437 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:00:57.129300 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:00:57.131042 (+ 1742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:57.131043 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:57.131044 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:57.131275 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:57.131500 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:57.131501 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:57.131501 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:57.131551 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:57.131892 (+ 341us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:00:57.131898 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:00:57.132021 (+ 123us) client_negotiation.cc:769] Sending connection context
0924 03:00:57.132107 (+ 86us) client_negotiation.cc:240] Negotiation successful
0924 03:00:57.132287 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":66}
I20260924 03:00:57.132498 17247 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:57.122725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35361 (local address 127.0.98.1:55295)
0924 03:00:57.122885 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:57.122892 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 03:00:57.122930 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:00:57.123709 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:00:57.123710 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:00:57.124095 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:00:57.124316 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:57.124322 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:57.125387 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:57.125388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:57.126193 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:00:57.126195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:00:57.126789 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:00:57.126789 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:00:57.126909 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:00:57.127486 (+ 577us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:00:57.127491 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:00:57.128895 (+ 1404us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:00:57.130838 (+ 1943us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:57.130844 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:57.130847 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:57.131174 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:57.131426 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:00:57.131426 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:00:57.131427 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:00:57.131542 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:00:57.131895 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:00:57.131902 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:00:57.132022 (+ 120us) client_negotiation.cc:769] Sending connection context
0924 03:00:57.132107 (+ 85us) client_negotiation.cc:240] Negotiation successful
0924 03:00:57.132360 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":174,"thread_start_us":71,"threads_started":1}
W20260924 03:00:57.133142 16860 leader_election.cc:341] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 69ed3bcd3eea43fbb1cedc5d3776bc39 (127.0.98.2:41077): Not found: Tablet not found: 4ed32090402f40e19b46d6fdbee8ee9d
W20260924 03:00:57.133755 16861 leader_election.cc:341] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361): Not found: Tablet not found: 4ed32090402f40e19b46d6fdbee8ee9d
I20260924 03:00:57.133801 16861 leader_election.cc:302] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9992277a03de478d9bcccf4c57b0629c; no voters: 237e991937424c92b84adf2579f7a2fa, 69ed3bcd3eea43fbb1cedc5d3776bc39
I20260924 03:00:57.133901 17246 raft_consensus.cc:2726] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:00:57.164152 17250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:57.163952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:60953)
0924 03:00:57.164070 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:57.164113 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":62,"threads_started":1}
I20260924 03:00:57.959960 17251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:57.959765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:55907)
0924 03:00:57.959878 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:57.959915 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
I20260924 03:00:58.038776 17252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:58.038573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:51731)
0924 03:00:58.038698 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:58.038738 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":63,"threads_started":1}
I20260924 03:00:58.164783 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:58.164559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:43195)
0924 03:00:58.164682 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:58.164723 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260924 03:00:58.703807 17254 raft_consensus.cc:482] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:00:58.703954 17254 raft_consensus.cc:504] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:00:58.704133 17254 leader_election.cc:288] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361), 69ed3bcd3eea43fbb1cedc5d3776bc39 (127.0.98.2:41077)
W20260924 03:00:58.704501 16861 leader_election.cc:341] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361): Not found: Tablet not found: 4ed32090402f40e19b46d6fdbee8ee9d
W20260924 03:00:58.704632 16860 leader_election.cc:341] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 69ed3bcd3eea43fbb1cedc5d3776bc39 (127.0.98.2:41077): Not found: Tablet not found: 4ed32090402f40e19b46d6fdbee8ee9d
I20260924 03:00:58.704690 16860 leader_election.cc:302] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9992277a03de478d9bcccf4c57b0629c; no voters: 237e991937424c92b84adf2579f7a2fa, 69ed3bcd3eea43fbb1cedc5d3776bc39
I20260924 03:00:58.704782 17254 raft_consensus.cc:2726] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:00:58.960618 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:58.960385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:59691)
0924 03:00:58.960530 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:58.960569 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":78,"threads_started":1}
I20260924 03:00:59.039364 17256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:59.039164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:35041)
0924 03:00:59.039290 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:59.039325 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260924 03:00:59.165365 17257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:59.165162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:57197)
0924 03:00:59.165291 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:59.165325 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260924 03:00:59.961295 17258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:00:59.961056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:34463)
0924 03:00:59.961183 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 03:00:59.961221 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":57,"threads_started":1}
I20260924 03:01:00.040005 17259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:00.039797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:44897)
0924 03:01:00.039920 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:00.039962 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260924 03:01:00.166087 17260 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:00.165839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:45815)
0924 03:01:00.165997 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:00.166039 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":82,"threads_started":1}
I20260924 03:01:00.815026 17261 raft_consensus.cc:482] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:01:00.815122 17261 raft_consensus.cc:504] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:01:00.815279 17261 leader_election.cc:288] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361), 69ed3bcd3eea43fbb1cedc5d3776bc39 (127.0.98.2:41077)
W20260924 03:01:00.815650 16861 leader_election.cc:341] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361): Not found: Tablet not found: 4ed32090402f40e19b46d6fdbee8ee9d
W20260924 03:01:00.815734 16860 leader_election.cc:341] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 69ed3bcd3eea43fbb1cedc5d3776bc39 (127.0.98.2:41077): Not found: Tablet not found: 4ed32090402f40e19b46d6fdbee8ee9d
I20260924 03:01:00.815773 16860 leader_election.cc:302] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9992277a03de478d9bcccf4c57b0629c; no voters: 237e991937424c92b84adf2579f7a2fa, 69ed3bcd3eea43fbb1cedc5d3776bc39
I20260924 03:01:00.815872 17261 raft_consensus.cc:2726] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:01:00.962042 17262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:00.961784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:57869)
0924 03:01:00.961937 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:00.961995 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":60,"threads_started":1}
I20260924 03:01:01.040648 17263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:01.040449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:40641)
0924 03:01:01.040561 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:01.040605 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
I20260924 03:01:01.166850 17264 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:01.166599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:37961)
0924 03:01:01.166757 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:01.166806 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":75,"threads_started":1}
I20260924 03:01:01.962783 17265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:01.962547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:40119)
0924 03:01:01.962692 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:01.962732 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":83,"threads_started":1}
I20260924 03:01:02.041312 17266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:02.041092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:45197)
0924 03:01:02.041227 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:02.041265 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260924 03:01:02.167507 17267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:02.167275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:53043)
0924 03:01:02.167424 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:02.167462 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":77,"threads_started":1}
I20260924 03:01:02.335747 17268 raft_consensus.cc:482] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:01:02.335841 17268 raft_consensus.cc:504] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:01:02.335978 17268 leader_election.cc:288] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361), 69ed3bcd3eea43fbb1cedc5d3776bc39 (127.0.98.2:41077)
W20260924 03:01:02.336364 16860 leader_election.cc:341] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 69ed3bcd3eea43fbb1cedc5d3776bc39 (127.0.98.2:41077): Not found: Tablet not found: 4ed32090402f40e19b46d6fdbee8ee9d
W20260924 03:01:02.336462 16861 leader_election.cc:341] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361): Not found: Tablet not found: 4ed32090402f40e19b46d6fdbee8ee9d
I20260924 03:01:02.336501 16861 leader_election.cc:302] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9992277a03de478d9bcccf4c57b0629c; no voters: 237e991937424c92b84adf2579f7a2fa, 69ed3bcd3eea43fbb1cedc5d3776bc39
I20260924 03:01:02.336583 17268 raft_consensus.cc:2726] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:01:02.963414 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:02.963192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:48945)
0924 03:01:02.963331 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:02.963369 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":63,"threads_started":1}
I20260924 03:01:03.041944 17270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:03.041746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:47307)
0924 03:01:03.041859 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:03.041901 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":61,"threads_started":1}
I20260924 03:01:03.168205 17271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:03.167965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:59121)
0924 03:01:03.168104 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:03.168145 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":70,"threads_started":1}
I20260924 03:01:03.964113 17272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:03.963892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:43885)
0924 03:01:03.964024 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:03.964066 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":62,"threads_started":1}
I20260924 03:01:04.042616 17273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:04.042376 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:44807)
0924 03:01:04.042500 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:04.042572 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":58,"threads_started":1}
I20260924 03:01:04.168831 17274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:04.168595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:44317)
0924 03:01:04.168724 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:04.168762 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":59,"threads_started":1}
I20260924 03:01:04.964819 17275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:04.964597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:45609)
0924 03:01:04.964732 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:04.964770 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260924 03:01:05.043273 17276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:05.043060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:52079)
0924 03:01:05.043176 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:05.043223 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":58,"threads_started":1}
I20260924 03:01:05.169510 17277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:05.169266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:57729)
0924 03:01:05.169421 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:05.169463 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":70,"threads_started":1}
I20260924 03:01:05.965440 17278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:05.965226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:40731)
0924 03:01:05.965358 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:05.965394 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":59,"threads_started":1}
I20260924 03:01:06.043903 17279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:06.043701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:52897)
0924 03:01:06.043815 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:06.043856 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":59,"threads_started":1}
I20260924 03:01:06.081075 17280 raft_consensus.cc:482] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:01:06.081166 17280 raft_consensus.cc:504] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "237e991937424c92b84adf2579f7a2fa" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35361 } } peers { permanent_uuid: "69ed3bcd3eea43fbb1cedc5d3776bc39" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41077 } } peers { permanent_uuid: "9992277a03de478d9bcccf4c57b0629c" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 38161 } }
I20260924 03:01:06.081322 17280 leader_election.cc:288] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361), 69ed3bcd3eea43fbb1cedc5d3776bc39 (127.0.98.2:41077)
W20260924 03:01:06.081677 16860 leader_election.cc:341] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 69ed3bcd3eea43fbb1cedc5d3776bc39 (127.0.98.2:41077): Not found: Tablet not found: 4ed32090402f40e19b46d6fdbee8ee9d
W20260924 03:01:06.081761 16861 leader_election.cc:341] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 237e991937424c92b84adf2579f7a2fa (127.0.98.3:35361): Not found: Tablet not found: 4ed32090402f40e19b46d6fdbee8ee9d
I20260924 03:01:06.081830 16861 leader_election.cc:302] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9992277a03de478d9bcccf4c57b0629c; no voters: 237e991937424c92b84adf2579f7a2fa, 69ed3bcd3eea43fbb1cedc5d3776bc39
I20260924 03:01:06.081907 17280 raft_consensus.cc:2726] T 4ed32090402f40e19b46d6fdbee8ee9d P 9992277a03de478d9bcccf4c57b0629c [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260924 03:01:06.170176 17281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:06.169954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:37381)
0924 03:01:06.170088 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:06.170130 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":62,"threads_started":1}
I20260924 03:01:06.966194 17282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:06.965939 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:41395)
0924 03:01:06.966105 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:06.966148 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":64,"threads_started":1}
I20260924 03:01:07.044579 17283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:07.044362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:50311)
0924 03:01:07.044494 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:07.044533 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20260924 03:01:07.170847 17284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:07.170626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:35641)
0924 03:01:07.170766 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:07.170801 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":73,"threads_started":1}
I20260924 03:01:07.966846 17285 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:07.966646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:52065)
0924 03:01:07.966761 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:07.966801 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":57,"threads_started":1}
I20260924 03:01:08.045212 17286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:08.045002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:51215)
0924 03:01:08.045129 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:08.045166 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260924 03:01:08.171458 17287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:08.171242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:44085)
0924 03:01:08.171378 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:08.171412 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260924 03:01:08.967468 17288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:08.967245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:55575)
0924 03:01:08.967385 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:08.967425 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":67,"threads_started":1}
I20260924 03:01:09.045872 17289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:09.045639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:54419)
0924 03:01:09.045775 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:09.045820 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":61,"threads_started":1}
I20260924 03:01:09.172058 17290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:09.171855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:51697)
0924 03:01:09.171983 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:09.172012 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":76,"threads_started":1}
I20260924 03:01:09.968065 17291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:09.967848 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:41971)
0924 03:01:09.967980 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:09.968017 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":60,"threads_started":1}
I20260924 03:01:10.046612 17292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:10.046381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:48009)
0924 03:01:10.046489 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:10.046566 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":60,"threads_started":1}
I20260924 03:01:10.172710 17293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:10.172499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:49167)
0924 03:01:10.172623 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:10.172664 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":71,"threads_started":1}
I20260924 03:01:10.968863 17294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:10.968657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.1:50351)
0924 03:01:10.968768 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:10.968815 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260924 03:01:11.047287 17295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:11.047055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.2:45083)
0924 03:01:11.047205 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:11.047242 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":69,"threads_started":1}
I20260924 03:01:11.167860 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 16813
I20260924 03:01:11.171962 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 16978
I20260924 03:01:11.173540 17296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:11.173358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37721 (local address 127.0.98.3:57449)
0924 03:01:11.173474 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:11.173500 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.62:37721: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":48,"threads_started":1}
I20260924 03:01:11.176614 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 17111
2026-09-24T03:01:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58000 ms)
[----------] 30 tests from SecurityITest (223846 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:11 dist-test-slave-sb6p krb5kdc[17299](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:11 dist-test-slave-sb6p krb5kdc[17299](info): set up 2 sockets
Sep 24 03:01:11 dist-test-slave-sb6p krb5kdc[17299](info): commencing operation
krb5kdc: starting...
W20260924 03:01:12.126013 392 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:12 dist-test-slave-sb6p krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:01:12.237913 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41705
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:41705
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:12.317097 17312 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:12.317303 17312 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:12.317332 17312 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:12.318475 17312 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:12.318542 17312 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:12.318564 17312 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:12.318588 17312 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:12.318611 17312 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:12.320070 17312 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:41705
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:41705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17312
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:12.320307 17312 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:12.320505 17312 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:12.325395 17320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:12.325479 17312 server_base.cc:1034] running on GCE node
W20260924 03:01:12.325546 17318 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:12.325620 17317 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:12.325994 17312 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:12.326251 17312 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:12.327396 17312 hybrid_clock.cc:648] HybridClock initialized: now 1790218872327375 us; error 39 us; skew 500 ppm
Sep 24 03:01:12 dist-test-slave-sb6p krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:12.329684 17312 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:01:12.330108 17312 webserver.cc:460] Webserver started at http://127.0.98.62:38519/ using document root <none> and password file <none>
I20260924 03:01:12.330300 17312 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:12.330343 17312 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:12.330458 17312 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:12.331498 17312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "a5031fa4507e4379bf5036ea09d1a831"
format_stamp: "Formatted at 2026-09-24 03:01:12 on dist-test-slave-sb6p"
server_key: "7067dcedffce40ef027c8e8effedcbe6"
server_key_iv: "42fe3302dd6d50858f6ca48ae9205abe"
server_key_version: "encryptionkey@0"
I20260924 03:01:12.331857 17312 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "a5031fa4507e4379bf5036ea09d1a831"
format_stamp: "Formatted at 2026-09-24 03:01:12 on dist-test-slave-sb6p"
server_key: "7067dcedffce40ef027c8e8effedcbe6"
server_key_iv: "42fe3302dd6d50858f6ca48ae9205abe"
server_key_version: "encryptionkey@0"
I20260924 03:01:12.333055 17312 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:01:12.333678 17327 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:12.333820 17312 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:12.333899 17312 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "a5031fa4507e4379bf5036ea09d1a831"
format_stamp: "Formatted at 2026-09-24 03:01:12 on dist-test-slave-sb6p"
server_key: "7067dcedffce40ef027c8e8effedcbe6"
server_key_iv: "42fe3302dd6d50858f6ca48ae9205abe"
server_key_version: "encryptionkey@0"
I20260924 03:01:12.333961 17312 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:12.347468 17312 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:12.349246 17312 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:12.349387 17312 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:12.353052 17312 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:41705
I20260924 03:01:12.353097 17379 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:41705 every 8 connection(s)
I20260924 03:01:12.353413 17312 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:12.354212 17380 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:12.356570 17380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831: Bootstrap starting.
I20260924 03:01:12.357127 17380 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:12.357375 17380 log.cc:824] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:12.358016 17380 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831: No bootstrap required, opened a new log
I20260924 03:01:12.359594 17380 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5031fa4507e4379bf5036ea09d1a831" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41705 } }
I20260924 03:01:12.359735 17380 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:12.359757 17380 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5031fa4507e4379bf5036ea09d1a831, State: Initialized, Role: FOLLOWER
I20260924 03:01:12.359894 17380 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5031fa4507e4379bf5036ea09d1a831" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41705 } }
I20260924 03:01:12.359972 17380 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:12.360005 17380 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:12.360051 17380 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:12.360531 17380 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5031fa4507e4379bf5036ea09d1a831" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41705 } }
I20260924 03:01:12.360615 17380 leader_election.cc:302] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5031fa4507e4379bf5036ea09d1a831; no voters:
I20260924 03:01:12.360764 17380 leader_election.cc:288] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:12.360800 17383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:12.360970 17383 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [term 1 LEADER]: Becoming Leader. State: Replica: a5031fa4507e4379bf5036ea09d1a831, State: Running, Role: LEADER
I20260924 03:01:12.360978 17380 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:12.361064 17383 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5031fa4507e4379bf5036ea09d1a831" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41705 } }
I20260924 03:01:12.361443 17385 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5031fa4507e4379bf5036ea09d1a831. Latest consensus state: current_term: 1 leader_uuid: "a5031fa4507e4379bf5036ea09d1a831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5031fa4507e4379bf5036ea09d1a831" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41705 } } }
I20260924 03:01:12.361527 17385 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:12.361567 17384 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5031fa4507e4379bf5036ea09d1a831" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5031fa4507e4379bf5036ea09d1a831" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 41705 } } }
I20260924 03:01:12.361637 17384 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:12.361759 17392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:12.362352 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 17312
I20260924 03:01:12.362418 17392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:12.362471 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:12.362764 392 external_mini_cluster.cc:1428] Setting key 5a4df6c7d5e46ac52856a4a4d5c7e1cc
I20260924 03:01:12.364089 17392 catalog_manager.cc:1347] Generated new cluster ID: cdc7daca26234b4e971617e95a75c449
I20260924 03:01:12.364138 17392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:01:12 dist-test-slave-sb6p krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:12.371729 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.363193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49996 (local address 127.0.98.62:41705)
0924 03:01:12.363331 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:12.363341 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:12.363419 (+ 78us) server_negotiation.cc:407] Connection header received
0924 03:01:12.363571 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:12.363578 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:12.363635 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:12.363762 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:12.364413 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.365323 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.366044 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.366479 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.366594 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.368780 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:12.368784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:12.368784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:12.368796 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:12.370267 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:12.370613 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:12.370616 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:12.370617 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:12.370715 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:12.370884 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:12.370884 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:12.370885 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:12.371136 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:12.371276 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:12.371371 (+ 95us) server_negotiation.cc:299] Negotiation successful
0924 03:01:12.371515 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":59,"threads_started":1}
I20260924 03:01:12.375115 17392 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:12.375819 17392 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:12.381335 17392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831: Generated new TSK 0
I20260924 03:01:12.381495 17392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:01:12.409153 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41705
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:12.503811 17406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:12.504006 17406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:12.504029 17406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:12.505224 17406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:12.505264 17406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:12.505290 17406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:01:12.506851 17406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17406
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:12.507071 17406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:12.507292 17406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:12.512013 17414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:12.512022 17411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:12.512140 17406 server_base.cc:1034] running on GCE node
W20260924 03:01:12.512085 17412 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:12.512439 17406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:12.512650 17406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:12.513794 17406 hybrid_clock.cc:648] HybridClock initialized: now 1790218872513775 us; error 32 us; skew 500 ppm
Sep 24 03:01:12 dist-test-slave-sb6p krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:12.516204 17406 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:01:12.516613 17406 webserver.cc:460] Webserver started at http://127.0.98.1:43147/ using document root <none> and password file <none>
I20260924 03:01:12.516798 17406 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:12.516834 17406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:12.516930 17406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:12.517858 17406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "4202d49846fd4842afb76bcb55c91129"
format_stamp: "Formatted at 2026-09-24 03:01:12 on dist-test-slave-sb6p"
server_key: "60f8cf4bf9e356508c7310cd23f82169"
server_key_iv: "f0b451280ba45f017efef83fcb68b007"
server_key_version: "encryptionkey@0"
I20260924 03:01:12.518137 17406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4202d49846fd4842afb76bcb55c91129"
format_stamp: "Formatted at 2026-09-24 03:01:12 on dist-test-slave-sb6p"
server_key: "60f8cf4bf9e356508c7310cd23f82169"
server_key_iv: "f0b451280ba45f017efef83fcb68b007"
server_key_version: "encryptionkey@0"
I20260924 03:01:12.519325 17406 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:01:12.520001 17421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:12.520154 17406 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:12.520234 17406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "4202d49846fd4842afb76bcb55c91129"
format_stamp: "Formatted at 2026-09-24 03:01:12 on dist-test-slave-sb6p"
server_key: "60f8cf4bf9e356508c7310cd23f82169"
server_key_iv: "f0b451280ba45f017efef83fcb68b007"
server_key_version: "encryptionkey@0"
I20260924 03:01:12.520287 17406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:12.530697 17406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:12.533279 17406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:12.533429 17406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:12.533665 17406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:12.534015 17406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:12.534061 17406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:12.534101 17406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:12.534125 17406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:12.540225 17406 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:43939
I20260924 03:01:12.540284 17534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:43939 every 8 connection(s)
I20260924 03:01:12.540580 17406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:01:12.544189 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 17406
I20260924 03:01:12.544310 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:01:12.544528 392 external_mini_cluster.cc:1428] Setting key 4ad2e561d3c97c7aa6593ae709d20b43
Sep 24 03:01:12 dist-test-slave-sb6p krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:12.550220 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.541240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:46793 (local address 127.0.98.62:41705)
0924 03:01:12.541290 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:12.541292 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:12.541529 (+ 237us) server_negotiation.cc:407] Connection header received
0924 03:01:12.542481 (+ 952us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:12.542483 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:12.542499 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:12.542577 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:12.543204 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.543707 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.544992 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.545251 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.545304 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.547557 (+ 2253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:12.547558 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:12.547559 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:12.547566 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:12.548869 (+ 1303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:12.549331 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:12.549332 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:12.549332 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:12.549379 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:12.549630 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:12.549631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:12.549631 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:12.549730 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:12.549775 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:12.550093 (+ 318us) server_negotiation.cc:299] Negotiation successful
0924 03:01:12.550153 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 03:01:12.550494 17537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.541219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41705 (local address 127.0.98.1:46793)
0924 03:01:12.541408 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:12.541446 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:12.542374 (+ 928us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:12.542654 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:12.542657 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:12.542921 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:12.543111 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:12.543115 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.543787 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.543788 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:12.544921 (+ 1133us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:12.544924 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.545321 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.545322 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:12.545453 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.545911 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:12.545919 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:12.547461 (+ 1542us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:12.548952 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:12.548959 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:12.548962 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:12.549260 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:12.549432 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:12.549433 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:12.549433 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:12.549571 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:12.549784 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:12.549789 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:12.549962 (+ 173us) client_negotiation.cc:769] Sending connection context
0924 03:01:12.550073 (+ 111us) client_negotiation.cc:240] Negotiation successful
0924 03:01:12.550251 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":72,"threads_started":1}
I20260924 03:01:12.551112 17535 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41705
I20260924 03:01:12.551215 17535 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:12.551422 17535 heartbeater.cc:511] Master 127.0.98.62:41705 requested a full tablet report, sending...
I20260924 03:01:12.551930 17344 ts_manager.cc:194] Registered new tserver with Master: 4202d49846fd4842afb76bcb55c91129 (127.0.98.1:43939)
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 03:01:12.552547 17344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:46793
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:01:12.568359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41705
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:12.646410 17540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:12.646672 17540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:12.646705 17540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:12.647869 17540 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:12.647908 17540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:12.647951 17540 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:01:12.649428 17540 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17540
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:12.649652 17540 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:12.649847 17540 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:12.654596 17545 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:12.654585 17548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:12.654652 17546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:12.654687 17540 server_base.cc:1034] running on GCE node
I20260924 03:01:12.655015 17540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:12.655221 17540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:12.656356 17540 hybrid_clock.cc:648] HybridClock initialized: now 1790218872656342 us; error 25 us; skew 500 ppm
Sep 24 03:01:12 dist-test-slave-sb6p krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:12.658648 17540 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:01:12.659076 17540 webserver.cc:460] Webserver started at http://127.0.98.2:46639/ using document root <none> and password file <none>
I20260924 03:01:12.659263 17540 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:12.659308 17540 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:12.659431 17540 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:12.660423 17540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "4410948f705642f3b2025ea8debc9282"
format_stamp: "Formatted at 2026-09-24 03:01:12 on dist-test-slave-sb6p"
server_key: "16c631eb7cf9604ecbf5d304f0c22463"
server_key_iv: "71a7024b9b5c8ec10a6b88ec216693ae"
server_key_version: "encryptionkey@0"
I20260924 03:01:12.660738 17540 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "4410948f705642f3b2025ea8debc9282"
format_stamp: "Formatted at 2026-09-24 03:01:12 on dist-test-slave-sb6p"
server_key: "16c631eb7cf9604ecbf5d304f0c22463"
server_key_iv: "71a7024b9b5c8ec10a6b88ec216693ae"
server_key_version: "encryptionkey@0"
I20260924 03:01:12.662072 17540 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:01:12.662760 17555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:12.662899 17540 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:12.662987 17540 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "4410948f705642f3b2025ea8debc9282"
format_stamp: "Formatted at 2026-09-24 03:01:12 on dist-test-slave-sb6p"
server_key: "16c631eb7cf9604ecbf5d304f0c22463"
server_key_iv: "71a7024b9b5c8ec10a6b88ec216693ae"
server_key_version: "encryptionkey@0"
I20260924 03:01:12.663048 17540 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:12.673729 17540 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:12.675422 17540 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:12.675545 17540 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:12.675751 17540 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:12.676033 17540 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:12.676066 17540 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:12.676086 17540 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:12.676110 17540 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:12.681382 17540 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43879
I20260924 03:01:12.681448 17668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43879 every 8 connection(s)
I20260924 03:01:12.681694 17540 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:01:12.682233 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 17540
I20260924 03:01:12.682334 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:01:12.682560 392 external_mini_cluster.cc:1428] Setting key 3cec1bc156d34a64e1dff92edae80e49
Sep 24 03:01:12 dist-test-slave-sb6p krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
I20260924 03:01:12.690953 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.682572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:41521 (local address 127.0.98.62:41705)
0924 03:01:12.682634 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:12.682637 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:12.682651 (+ 14us) server_negotiation.cc:407] Connection header received
0924 03:01:12.683530 (+ 879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:12.683531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:12.683542 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:12.683602 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:12.684161 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.684705 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.685824 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.686118 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.686189 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.688506 (+ 2317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:12.688507 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:12.688507 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:12.688514 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:12.689735 (+ 1221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:12.690123 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:12.690124 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:12.690124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:12.690171 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:12.690347 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:12.690348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:12.690348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:12.690463 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:12.690508 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:12.690810 (+ 302us) server_negotiation.cc:299] Negotiation successful
0924 03:01:12.690884 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:01:12.691115 17671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.682316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41705 (local address 127.0.98.2:41521)
0924 03:01:12.682456 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:12.682496 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:12.683426 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:12.683655 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:12.683659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:12.683897 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:12.684089 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:12.684094 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.684781 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.684783 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:12.685743 (+ 960us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:12.685746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.686192 (+ 446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.686193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:12.686292 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.686804 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:12.686812 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:12.688420 (+ 1608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:12.689800 (+ 1380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:12.689807 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:12.689809 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:12.690066 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:12.690220 (+ 154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:12.690221 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:12.690221 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:12.690308 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:12.690505 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:12.690507 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:12.690656 (+ 149us) client_negotiation.cc:769] Sending connection context
0924 03:01:12.690786 (+ 130us) client_negotiation.cc:240] Negotiation successful
0924 03:01:12.690909 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":64,"threads_started":1}
I20260924 03:01:12.691522 17669 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41705
I20260924 03:01:12.691612 17669 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:12.691787 17669 heartbeater.cc:511] Master 127.0.98.62:41705 requested a full tablet report, sending...
I20260924 03:01:12.692119 17344 ts_manager.cc:194] Registered new tserver with Master: 4410948f705642f3b2025ea8debc9282 (127.0.98.2:43879)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
I20260924 03:01:12.692569 17344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:41521
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:01:12.705190 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41705
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44679
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:12.783789 17674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:12.783993 17674 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:12.784015 17674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:12.785138 17674 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:12.785177 17674 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:12.785205 17674 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:01:12.786659 17674 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44679
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:41705
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17674
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:12.786864 17674 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:12.787070 17674 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:12.791703 17679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:12.791841 17674 server_base.cc:1034] running on GCE node
W20260924 03:01:12.791703 17680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:12.791970 17682 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:12.792146 17674 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:12.792356 17674 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:12.793488 17674 hybrid_clock.cc:648] HybridClock initialized: now 1790218872793484 us; error 35 us; skew 500 ppm
Sep 24 03:01:12 dist-test-slave-sb6p krb5kdc[17299](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:12.795765 17674 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:01:12.796156 17674 webserver.cc:460] Webserver started at http://127.0.98.3:37181/ using document root <none> and password file <none>
I20260924 03:01:12.796348 17674 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:12.796393 17674 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:12.796523 17674 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:12.797516 17674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "dbbdb73fd42a47849e322902970cd61c"
format_stamp: "Formatted at 2026-09-24 03:01:12 on dist-test-slave-sb6p"
server_key: "fef4e1d6e0b0c11af05e7901c44a8a61"
server_key_iv: "26f77046fd44a1a7f3f322434daf3227"
server_key_version: "encryptionkey@0"
I20260924 03:01:12.797833 17674 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "dbbdb73fd42a47849e322902970cd61c"
format_stamp: "Formatted at 2026-09-24 03:01:12 on dist-test-slave-sb6p"
server_key: "fef4e1d6e0b0c11af05e7901c44a8a61"
server_key_iv: "26f77046fd44a1a7f3f322434daf3227"
server_key_version: "encryptionkey@0"
I20260924 03:01:12.799060 17674 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:01:12.799698 17689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:12.799834 17674 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:12.799921 17674 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "dbbdb73fd42a47849e322902970cd61c"
format_stamp: "Formatted at 2026-09-24 03:01:12 on dist-test-slave-sb6p"
server_key: "fef4e1d6e0b0c11af05e7901c44a8a61"
server_key_iv: "26f77046fd44a1a7f3f322434daf3227"
server_key_version: "encryptionkey@0"
I20260924 03:01:12.799980 17674 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:12.816833 17674 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:12.818878 17674 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:12.819007 17674 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:12.819217 17674 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:12.819490 17674 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:12.819521 17674 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:12.819550 17674 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:12.819573 17674 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:12.825387 17674 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:39481
I20260924 03:01:12.825444 17802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:39481 every 8 connection(s)
I20260924 03:01:12.825721 17674 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:01:12.829267 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 17674
I20260924 03:01:12.829360 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:01:12.829543 392 external_mini_cluster.cc:1428] Setting key d4decbfcca9aeb30da74532bee60a04b
Sep 24 03:01:12 dist-test-slave-sb6p krb5kdc[17299](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:12.834555 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.826359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:52221 (local address 127.0.98.62:41705)
0924 03:01:12.826424 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:12.826426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:12.826629 (+ 203us) server_negotiation.cc:407] Connection header received
0924 03:01:12.827549 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:12.827551 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:12.827562 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:12.827613 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:12.828207 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.828706 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.829762 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.830059 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.830127 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.832178 (+ 2051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:12.832180 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:12.832180 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:12.832187 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:12.833265 (+ 1078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:12.833671 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:12.833672 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:12.833673 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:12.833730 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:12.833943 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:12.833944 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:12.833945 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:12.834060 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:12.834114 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:12.834361 (+ 247us) server_negotiation.cc:299] Negotiation successful
0924 03:01:12.834437 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 03:01:12.834743 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.826326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:41705 (local address 127.0.98.3:52221)
0924 03:01:12.826483 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:12.826540 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:12.827438 (+ 898us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:12.827683 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:12.827686 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:12.827925 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:12.828112 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:12.828116 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.828789 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.828790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:12.829677 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:12.829680 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.830147 (+ 467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.830148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:12.830294 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.830818 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:12.830827 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:12.832101 (+ 1274us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:12.833346 (+ 1245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:12.833352 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:12.833354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:12.833577 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:12.833784 (+ 207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:12.833785 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:12.833785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:12.833879 (+ 94us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:12.834113 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:12.834119 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:12.834247 (+ 128us) client_negotiation.cc:769] Sending connection context
0924 03:01:12.834334 (+ 87us) client_negotiation.cc:240] Negotiation successful
0924 03:01:12.834494 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260924 03:01:12.835129 17803 heartbeater.cc:347] Connected to a master server at 127.0.98.62:41705
I20260924 03:01:12.835219 17803 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:12.835381 17803 heartbeater.cc:511] Master 127.0.98.62:41705 requested a full tablet report, sending...
I20260924 03:01:12.835675 17344 ts_manager.cc:194] Registered new tserver with Master: dbbdb73fd42a47849e322902970cd61c (127.0.98.3:39481)
I20260924 03:01:12.836019 17344 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:52221
I20260924 03:01:12.841121 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:01:12.847611 17402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.842382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50010 (local address 127.0.98.62:41705)
0924 03:01:12.842422 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:12.842424 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:12.842494 (+ 70us) server_negotiation.cc:407] Connection header received
0924 03:01:12.842588 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:12.842588 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:12.842597 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:12.842639 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:12.843137 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.843614 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.844335 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.844518 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.844560 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.845582 (+ 1022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:12.845583 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:12.845583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:12.845588 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:12.846787 (+ 1199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:12.847071 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:12.847072 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:12.847072 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:12.847112 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:12.847236 (+ 124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:12.847236 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:12.847237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:12.847355 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:12.847422 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:12.847498 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 03:01:12.847551 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:01:12.848901 17344 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50010:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:01:12.850392 17344 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:01:12.856496 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.852485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43939 (local address 127.0.0.1:35422)
0924 03:01:12.852809 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:12.852816 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:12.852852 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:12.853138 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:12.853139 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:12.853141 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:12.853182 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:12.853184 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.854358 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.854359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:12.855332 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:12.855334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.856205 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.856206 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:12.856277 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.856290 (+ 13us) client_negotiation.cc:769] Sending connection context
0924 03:01:12.856356 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 03:01:12.856405 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":314,"thread_start_us":61,"threads_started":1}
I20260924 03:01:12.856510 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.852618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35422 (local address 127.0.98.1:43939)
0924 03:01:12.852928 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:12.852931 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:12.852940 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:01:12.852978 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:12.852980 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:12.852994 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:12.853059 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:12.853254 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.854282 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.855407 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.856122 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.856198 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.856261 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:12.856383 (+ 122us) server_negotiation.cc:299] Negotiation successful
0924 03:01:12.856426 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":67,"threads_started":1}
I20260924 03:01:12.856643 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.852401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35510 (local address 127.0.98.3:39481)
0924 03:01:12.852693 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:12.852698 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:12.852810 (+ 112us) server_negotiation.cc:407] Connection header received
0924 03:01:12.852877 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:12.852879 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:12.852894 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:12.852977 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:12.853116 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.854210 (+ 1094us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.855513 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.856160 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.856213 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.856261 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:12.856537 (+ 276us) server_negotiation.cc:299] Negotiation successful
0924 03:01:12.856574 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":258,"thread_start_us":100,"threads_started":1}
I20260924 03:01:12.856691 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.852294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:39481 (local address 127.0.0.1:35510)
0924 03:01:12.852745 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:12.852754 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:12.852812 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:12.852982 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:12.852982 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:12.852984 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:12.853054 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:12.853056 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.854291 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.854292 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:12.855441 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:12.855444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.856294 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.856295 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:12.856366 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.856369 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:12.856415 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 03:01:12.856454 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":432,"mutex_wait_us":30,"thread_start_us":62,"threads_started":1}
I20260924 03:01:12.856765 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.852490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49616 (local address 127.0.98.2:43879)
0924 03:01:12.852647 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:12.852652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:12.852689 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:01:12.852787 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:12.852790 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:12.852805 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:12.852874 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:12.853218 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.853992 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.855697 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.856165 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.856228 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.856284 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:12.856494 (+ 210us) server_negotiation.cc:299] Negotiation successful
0924 03:01:12.856532 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260924 03:01:12.856870 17811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.852279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43879 (local address 127.0.0.1:49616)
0924 03:01:12.852598 (+ 319us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:12.852640 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:12.852710 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:12.852977 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:12.852980 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:12.852982 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:12.853144 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:12.853146 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.854135 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.854136 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:12.855635 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:12.855637 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.856312 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.856313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:12.856367 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.856370 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:12.856414 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 03:01:12.856445 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":293,"mutex_wait_us":4,"thread_start_us":73,"threads_started":1}
I20260924 03:01:12.857398 17737 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec1d40349794f22870a48bb8882fa50 (DEFAULT_TABLE table=test-table [id=e881297875724b768726c7226319b8d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:12.857398 17469 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec1d40349794f22870a48bb8882fa50 (DEFAULT_TABLE table=test-table [id=e881297875724b768726c7226319b8d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:12.857559 17603 tablet_service.cc:1467] Processing CreateTablet for tablet 5ec1d40349794f22870a48bb8882fa50 (DEFAULT_TABLE table=test-table [id=e881297875724b768726c7226319b8d7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:12.857734 17469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec1d40349794f22870a48bb8882fa50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:12.857733 17737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec1d40349794f22870a48bb8882fa50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:12.857824 17603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5ec1d40349794f22870a48bb8882fa50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:12.860791 17819 tablet_bootstrap.cc:492] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282: Bootstrap starting.
I20260924 03:01:12.861076 17817 tablet_bootstrap.cc:492] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c: Bootstrap starting.
I20260924 03:01:12.861469 17819 tablet_bootstrap.cc:654] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:12.861609 17817 tablet_bootstrap.cc:654] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:12.862072 17819 log.cc:824] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:12.862079 17817 log.cc:824] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:12.862376 17818 tablet_bootstrap.cc:492] T 5ec1d40349794f22870a48bb8882fa50 P 4202d49846fd4842afb76bcb55c91129: Bootstrap starting.
I20260924 03:01:12.862738 17819 tablet_bootstrap.cc:492] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282: No bootstrap required, opened a new log
I20260924 03:01:12.862756 17817 tablet_bootstrap.cc:492] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c: No bootstrap required, opened a new log
I20260924 03:01:12.862811 17817 ts_tablet_manager.cc:1397] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:01:12.862804 17819 ts_tablet_manager.cc:1397] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:01:12.864228 17817 raft_consensus.cc:348] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4410948f705642f3b2025ea8debc9282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "dbbdb73fd42a47849e322902970cd61c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39481 } } peers { permanent_uuid: "4202d49846fd4842afb76bcb55c91129" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43939 } }
I20260924 03:01:12.864257 17819 raft_consensus.cc:348] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4410948f705642f3b2025ea8debc9282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "dbbdb73fd42a47849e322902970cd61c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39481 } } peers { permanent_uuid: "4202d49846fd4842afb76bcb55c91129" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43939 } }
I20260924 03:01:12.864360 17817 raft_consensus.cc:374] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:12.864382 17817 raft_consensus.cc:729] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dbbdb73fd42a47849e322902970cd61c, State: Initialized, Role: FOLLOWER
I20260924 03:01:12.864375 17819 raft_consensus.cc:374] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:12.864408 17819 raft_consensus.cc:729] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4410948f705642f3b2025ea8debc9282, State: Initialized, Role: FOLLOWER
I20260924 03:01:12.864508 17817 consensus_queue.cc:260] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4410948f705642f3b2025ea8debc9282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "dbbdb73fd42a47849e322902970cd61c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39481 } } peers { permanent_uuid: "4202d49846fd4842afb76bcb55c91129" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43939 } }
I20260924 03:01:12.864513 17819 consensus_queue.cc:260] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4410948f705642f3b2025ea8debc9282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "dbbdb73fd42a47849e322902970cd61c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39481 } } peers { permanent_uuid: "4202d49846fd4842afb76bcb55c91129" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43939 } }
I20260924 03:01:12.864759 17817 ts_tablet_manager.cc:1428] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:01:12.864763 17819 ts_tablet_manager.cc:1428] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260924 03:01:12.864768 17669 heartbeater.cc:503] Master 127.0.98.62:41705 was elected leader, sending a full tablet report...
I20260924 03:01:12.864843 17803 heartbeater.cc:503] Master 127.0.98.62:41705 was elected leader, sending a full tablet report...
I20260924 03:01:12.865407 17818 tablet_bootstrap.cc:654] T 5ec1d40349794f22870a48bb8882fa50 P 4202d49846fd4842afb76bcb55c91129: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:12.865976 17818 log.cc:824] T 5ec1d40349794f22870a48bb8882fa50 P 4202d49846fd4842afb76bcb55c91129: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:12.866587 17818 tablet_bootstrap.cc:492] T 5ec1d40349794f22870a48bb8882fa50 P 4202d49846fd4842afb76bcb55c91129: No bootstrap required, opened a new log
I20260924 03:01:12.866639 17818 ts_tablet_manager.cc:1397] T 5ec1d40349794f22870a48bb8882fa50 P 4202d49846fd4842afb76bcb55c91129: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260924 03:01:12.868124 17818 raft_consensus.cc:348] T 5ec1d40349794f22870a48bb8882fa50 P 4202d49846fd4842afb76bcb55c91129 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4410948f705642f3b2025ea8debc9282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "dbbdb73fd42a47849e322902970cd61c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39481 } } peers { permanent_uuid: "4202d49846fd4842afb76bcb55c91129" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43939 } }
I20260924 03:01:12.868330 17818 raft_consensus.cc:374] T 5ec1d40349794f22870a48bb8882fa50 P 4202d49846fd4842afb76bcb55c91129 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:12.868381 17818 raft_consensus.cc:729] T 5ec1d40349794f22870a48bb8882fa50 P 4202d49846fd4842afb76bcb55c91129 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4202d49846fd4842afb76bcb55c91129, State: Initialized, Role: FOLLOWER
I20260924 03:01:12.868487 17818 consensus_queue.cc:260] T 5ec1d40349794f22870a48bb8882fa50 P 4202d49846fd4842afb76bcb55c91129 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4410948f705642f3b2025ea8debc9282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "dbbdb73fd42a47849e322902970cd61c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39481 } } peers { permanent_uuid: "4202d49846fd4842afb76bcb55c91129" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43939 } }
I20260924 03:01:12.868716 17818 ts_tablet_manager.cc:1428] T 5ec1d40349794f22870a48bb8882fa50 P 4202d49846fd4842afb76bcb55c91129: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:01:12.868743 17535 heartbeater.cc:503] Master 127.0.98.62:41705 was elected leader, sending a full tablet report...
I20260924 03:01:12.922019 17822 raft_consensus.cc:482] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:01:12.922150 17822 raft_consensus.cc:504] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4410948f705642f3b2025ea8debc9282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "dbbdb73fd42a47849e322902970cd61c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39481 } } peers { permanent_uuid: "4202d49846fd4842afb76bcb55c91129" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43939 } }
I20260924 03:01:12.922573 17822 leader_election.cc:288] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4410948f705642f3b2025ea8debc9282 (127.0.98.2:43879), 4202d49846fd4842afb76bcb55c91129 (127.0.98.1:43939)
I20260924 03:01:12.925329 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.922702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43879 (local address 127.0.98.3:33729)
0924 03:01:12.922759 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:12.922773 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:12.922835 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:12.922986 (+ 151us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:12.922987 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:12.922988 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:12.923074 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:12.923076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.923730 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.923731 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:12.924595 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:12.924598 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.925099 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.925100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:12.925171 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.925173 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:01:12.925215 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 03:01:12.925250 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260924 03:01:12.925325 17815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.922830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33729 (local address 127.0.98.2:43879)
0924 03:01:12.922881 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:12.922883 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:12.922891 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:01:12.922918 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:12.922919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:12.922929 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:12.922975 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:12.923136 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.923660 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.924675 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.925029 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.925086 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.925097 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:12.925224 (+ 127us) server_negotiation.cc:299] Negotiation successful
0924 03:01:12.925259 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:01:12.925442 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.922771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:43717 (local address 127.0.98.1:43939)
0924 03:01:12.922829 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:12.922831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:12.922953 (+ 122us) server_negotiation.cc:407] Connection header received
0924 03:01:12.923003 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:12.923004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:12.923017 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:12.923060 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:12.923253 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.923763 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.924771 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.925132 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.925190 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.925200 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:12.925337 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 03:01:12.925375 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 03:01:12.925458 17826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.922702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43939 (local address 127.0.98.3:43717)
0924 03:01:12.922877 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:12.922886 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:12.922947 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:12.923125 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:12.923126 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:12.923128 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:12.923190 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:12.923191 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.923821 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.923822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:12.924687 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:12.924690 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.925202 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.925202 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:12.925281 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.925284 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:12.925328 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 03:01:12.925364 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":13,"thread_start_us":88,"threads_started":1}
I20260924 03:01:12.925661 17623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec1d40349794f22870a48bb8882fa50" candidate_uuid: "dbbdb73fd42a47849e322902970cd61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4410948f705642f3b2025ea8debc9282" is_pre_election: true
I20260924 03:01:12.925777 17489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec1d40349794f22870a48bb8882fa50" candidate_uuid: "dbbdb73fd42a47849e322902970cd61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4202d49846fd4842afb76bcb55c91129" is_pre_election: true
I20260924 03:01:12.925832 17623 raft_consensus.cc:2454] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbbdb73fd42a47849e322902970cd61c in term 0.
I20260924 03:01:12.925904 17489 raft_consensus.cc:2454] T 5ec1d40349794f22870a48bb8882fa50 P 4202d49846fd4842afb76bcb55c91129 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dbbdb73fd42a47849e322902970cd61c in term 0.
I20260924 03:01:12.926106 17691 leader_election.cc:302] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4202d49846fd4842afb76bcb55c91129, dbbdb73fd42a47849e322902970cd61c; no voters:
I20260924 03:01:12.926249 17822 raft_consensus.cc:2781] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:01:12.926303 17822 raft_consensus.cc:482] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:01:12.926319 17822 raft_consensus.cc:3037] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:12.927031 17822 raft_consensus.cc:504] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4410948f705642f3b2025ea8debc9282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "dbbdb73fd42a47849e322902970cd61c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39481 } } peers { permanent_uuid: "4202d49846fd4842afb76bcb55c91129" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43939 } }
I20260924 03:01:12.927143 17822 leader_election.cc:288] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [CANDIDATE]: Term 1 election: Requested vote from peers 4410948f705642f3b2025ea8debc9282 (127.0.98.2:43879), 4202d49846fd4842afb76bcb55c91129 (127.0.98.1:43939)
I20260924 03:01:12.927294 17623 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec1d40349794f22870a48bb8882fa50" candidate_uuid: "dbbdb73fd42a47849e322902970cd61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4410948f705642f3b2025ea8debc9282"
I20260924 03:01:12.927331 17489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5ec1d40349794f22870a48bb8882fa50" candidate_uuid: "dbbdb73fd42a47849e322902970cd61c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4202d49846fd4842afb76bcb55c91129"
I20260924 03:01:12.927394 17489 raft_consensus.cc:3037] T 5ec1d40349794f22870a48bb8882fa50 P 4202d49846fd4842afb76bcb55c91129 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:12.927388 17623 raft_consensus.cc:3037] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:12.928238 17623 raft_consensus.cc:2454] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbbdb73fd42a47849e322902970cd61c in term 1.
I20260924 03:01:12.928238 17489 raft_consensus.cc:2454] T 5ec1d40349794f22870a48bb8882fa50 P 4202d49846fd4842afb76bcb55c91129 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dbbdb73fd42a47849e322902970cd61c in term 1.
I20260924 03:01:12.928419 17691 leader_election.cc:302] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4202d49846fd4842afb76bcb55c91129, dbbdb73fd42a47849e322902970cd61c; no voters:
I20260924 03:01:12.928489 17822 raft_consensus.cc:2781] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:12.928615 17822 raft_consensus.cc:686] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [term 1 LEADER]: Becoming Leader. State: Replica: dbbdb73fd42a47849e322902970cd61c, State: Running, Role: LEADER
I20260924 03:01:12.928694 17822 consensus_queue.cc:237] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4410948f705642f3b2025ea8debc9282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } } peers { permanent_uuid: "dbbdb73fd42a47849e322902970cd61c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39481 } } peers { permanent_uuid: "4202d49846fd4842afb76bcb55c91129" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43939 } }
I20260924 03:01:12.929562 17343 catalog_manager.cc:5495] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c reported cstate change: term changed from 0 to 1, leader changed from <none> to dbbdb73fd42a47849e322902970cd61c (127.0.98.3). New cstate: current_term: 1 leader_uuid: "dbbdb73fd42a47849e322902970cd61c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4410948f705642f3b2025ea8debc9282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dbbdb73fd42a47849e322902970cd61c" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 39481 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4202d49846fd4842afb76bcb55c91129" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43939 } health_report { overall_health: UNKNOWN } } }
W20260924 03:01:12.932159 17670 tablet.cc:2369] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:01:12.946331 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.944273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35524 (local address 127.0.98.3:39481)
0924 03:01:12.944324 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:12.944326 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:12.944334 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:01:12.944366 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:12.944366 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:12.944382 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:12.944449 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:01:12.944551 (+ 102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.945005 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.945736 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:12.945925 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:12.945976 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:12.946079 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:01:12.946172 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:01:12.946227 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:12.946242 (+ 15us) server_negotiation.cc:299] Negotiation successful
0924 03:01:12.946258 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260924 03:01:12.948066 17623 raft_consensus.cc:1261] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282 [term 1 FOLLOWER]: Refusing update from remote peer dbbdb73fd42a47849e322902970cd61c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:01:12.948076 17489 raft_consensus.cc:1261] T 5ec1d40349794f22870a48bb8882fa50 P 4202d49846fd4842afb76bcb55c91129 [term 1 FOLLOWER]: Refusing update from remote peer dbbdb73fd42a47849e322902970cd61c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:01:12.948360 17822 consensus_queue.cc:1035] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4410948f705642f3b2025ea8debc9282" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:01:12.948485 17827 consensus_queue.cc:1035] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [LEADER]: Connected to new peer: Peer: permanent_uuid: "4202d49846fd4842afb76bcb55c91129" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 43939 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:01:12.950671 17830 mvcc.cc:205] Tried to move back new op lower bound from 7332736503592927232 to 7332736503516680192. Current Snapshot: MvccSnapshot[applied={T|T < 7332736503592927232}]
I20260924 03:01:12.950834 17832 mvcc.cc:205] Tried to move back new op lower bound from 7332736503592927232 to 7332736503516680192. Current Snapshot: MvccSnapshot[applied={T|T < 7332736503592927232}]
I20260924 03:01:12.952967 17343 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:01:12.953049 17343 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50010:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:01:12.953724 17343 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5031fa4507e4379bf5036ea09d1a831: Sending DeleteTablet for 3 replicas of tablet 5ec1d40349794f22870a48bb8882fa50
I20260924 03:01:12.954051 17737 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ec1d40349794f22870a48bb8882fa50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35510
I20260924 03:01:12.954080 17469 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ec1d40349794f22870a48bb8882fa50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:35422
I20260924 03:01:12.954288 17603 tablet_service.cc:1514] Processing DeleteTablet for tablet 5ec1d40349794f22870a48bb8882fa50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:12 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:49616
I20260924 03:01:12.954463 17841 tablet_replica.cc:330] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282: stopping tablet replica
I20260924 03:01:12.954576 17841 raft_consensus.cc:2229] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:01:12.954672 17841 raft_consensus.cc:2258] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:01:12.954814 17840 tablet_replica.cc:330] T 5ec1d40349794f22870a48bb8882fa50 P 4202d49846fd4842afb76bcb55c91129: stopping tablet replica
I20260924 03:01:12.954916 17840 raft_consensus.cc:2229] T 5ec1d40349794f22870a48bb8882fa50 P 4202d49846fd4842afb76bcb55c91129 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:01:12.955013 17840 raft_consensus.cc:2258] T 5ec1d40349794f22870a48bb8882fa50 P 4202d49846fd4842afb76bcb55c91129 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:01:12.955036 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 17406
I20260924 03:01:12.955822 17839 tablet_replica.cc:330] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c: stopping tablet replica
I20260924 03:01:12.955916 17839 raft_consensus.cc:2229] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:01:12.956008 17839 raft_consensus.cc:2258] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:01:12.956276 17841 ts_tablet_manager.cc:1905] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:01:12.956776 17839 ts_tablet_manager.cc:1905] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:01:12.958051 17841 ts_tablet_manager.cc:1918] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:01:12.958120 17841 log.cc:1196] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/5ec1d40349794f22870a48bb8882fa50
I20260924 03:01:12.958351 17841 ts_tablet_manager.cc:1939] T 5ec1d40349794f22870a48bb8882fa50 P 4410948f705642f3b2025ea8debc9282: Deleting consensus metadata
I20260924 03:01:12.958508 17839 ts_tablet_manager.cc:1918] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:01:12.958642 17839 log.cc:1196] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/5ec1d40349794f22870a48bb8882fa50
I20260924 03:01:12.958739 17331 catalog_manager.cc:4841] TS 4410948f705642f3b2025ea8debc9282 (127.0.98.2:43879): tablet 5ec1d40349794f22870a48bb8882fa50 (table test-table [id=e881297875724b768726c7226319b8d7]) successfully deleted
I20260924 03:01:12.959074 17839 ts_tablet_manager.cc:1939] T 5ec1d40349794f22870a48bb8882fa50 P dbbdb73fd42a47849e322902970cd61c: Deleting consensus metadata
I20260924 03:01:12.959457 17329 catalog_manager.cc:4841] TS dbbdb73fd42a47849e322902970cd61c (127.0.98.3:39481): tablet 5ec1d40349794f22870a48bb8882fa50 (table test-table [id=e881297875724b768726c7226319b8d7]) successfully deleted
W20260924 03:01:12.960634 17329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:43939 (error 108)
I20260924 03:01:12.960896 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 17540
I20260924 03:01:12.961424 17811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:12.960950 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:43939 (local address 127.0.0.1:35428)
0924 03:01:12.961352 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:12.961392 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43939: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":385}
W20260924 03:01:12.961730 17329 catalog_manager.cc:4584] TS 4202d49846fd4842afb76bcb55c91129 (127.0.98.1:43939): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 5ec1d40349794f22870a48bb8882fa50: Network error: Client connection negotiation failed: client connection to 127.0.98.1:43939: connect: Connection refused (error 111)
I20260924 03:01:12.965801 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 17674
I20260924 03:01:12.969969 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 17312
2026-09-24T03:01:12Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1793 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-24T03:01:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:12Z Disabled control of system clock
I20260924 03:01:12.995133 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37601
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37601
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:01:13.072381 17846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:13.072589 17846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:13.072608 17846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:13.073679 17846 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:13.073711 17846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:13.073724 17846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:13.073734 17846 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:13.073745 17846 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:13.075037 17846 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37601
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37601
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17846
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:13.075218 17846 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:13.075448 17846 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:13.079952 17851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:13.079952 17854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:13.080101 17846 server_base.cc:1034] running on GCE node
W20260924 03:01:13.080041 17852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:13.080351 17846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:13.080619 17846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:13.081755 17846 hybrid_clock.cc:648] HybridClock initialized: now 1790218873081743 us; error 24 us; skew 500 ppm
I20260924 03:01:13.082872 17846 webserver.cc:460] Webserver started at http://127.0.98.62:44139/ using document root <none> and password file <none>
I20260924 03:01:13.083086 17846 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:13.083134 17846 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:13.083253 17846 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:13.084252 17846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "cadc472948db491c8b089183f978a3cc"
format_stamp: "Formatted at 2026-09-24 03:01:13 on dist-test-slave-sb6p"
server_key: "1c3845ac649a0c5b08e7fcd93d88298f"
server_key_iv: "2bd9a7810ce85de21cbed379be7d12bf"
server_key_version: "encryptionkey@0"
I20260924 03:01:13.084565 17846 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "cadc472948db491c8b089183f978a3cc"
format_stamp: "Formatted at 2026-09-24 03:01:13 on dist-test-slave-sb6p"
server_key: "1c3845ac649a0c5b08e7fcd93d88298f"
server_key_iv: "2bd9a7810ce85de21cbed379be7d12bf"
server_key_version: "encryptionkey@0"
I20260924 03:01:13.085713 17846 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:01:13.086337 17860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:13.086494 17846 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:13.086606 17846 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "cadc472948db491c8b089183f978a3cc"
format_stamp: "Formatted at 2026-09-24 03:01:13 on dist-test-slave-sb6p"
server_key: "1c3845ac649a0c5b08e7fcd93d88298f"
server_key_iv: "2bd9a7810ce85de21cbed379be7d12bf"
server_key_version: "encryptionkey@0"
I20260924 03:01:13.086668 17846 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:13.098258 17846 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:13.098554 17846 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:13.098686 17846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:13.102119 17846 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37601
I20260924 03:01:13.102149 17912 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37601 every 8 connection(s)
I20260924 03:01:13.102447 17846 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:13.103390 17913 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:13.105505 17913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc: Bootstrap starting.
I20260924 03:01:13.106021 17913 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:13.106252 17913 log.cc:824] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:13.106856 17913 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc: No bootstrap required, opened a new log
I20260924 03:01:13.108382 17913 raft_consensus.cc:348] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadc472948db491c8b089183f978a3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37601 } }
I20260924 03:01:13.108511 17913 raft_consensus.cc:374] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:13.108532 17913 raft_consensus.cc:729] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cadc472948db491c8b089183f978a3cc, State: Initialized, Role: FOLLOWER
I20260924 03:01:13.108661 17913 consensus_queue.cc:260] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadc472948db491c8b089183f978a3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37601 } }
I20260924 03:01:13.108731 17913 raft_consensus.cc:388] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:13.108752 17913 raft_consensus.cc:482] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:13.108801 17913 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:13.108793 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 17846
I20260924 03:01:13.108896 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:13.109081 392 external_mini_cluster.cc:1428] Setting key 36126f864eb0267122cdd6f317a203a5
I20260924 03:01:13.109357 17913 raft_consensus.cc:504] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadc472948db491c8b089183f978a3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37601 } }
I20260924 03:01:13.109459 17913 leader_election.cc:302] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cadc472948db491c8b089183f978a3cc; no voters:
I20260924 03:01:13.109627 17913 leader_election.cc:288] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:13.109704 17917 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:13.109825 17917 raft_consensus.cc:686] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [term 1 LEADER]: Becoming Leader. State: Replica: cadc472948db491c8b089183f978a3cc, State: Running, Role: LEADER
I20260924 03:01:13.109892 17913 sys_catalog.cc:564] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:13.109911 17917 consensus_queue.cc:237] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadc472948db491c8b089183f978a3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37601 } }
I20260924 03:01:13.110224 17917 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader cadc472948db491c8b089183f978a3cc. Latest consensus state: current_term: 1 leader_uuid: "cadc472948db491c8b089183f978a3cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadc472948db491c8b089183f978a3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37601 } } }
I20260924 03:01:13.110299 17917 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:13.110556 17920 sys_catalog.cc:455] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cadc472948db491c8b089183f978a3cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cadc472948db491c8b089183f978a3cc" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37601 } } }
I20260924 03:01:13.110687 17920 sys_catalog.cc:458] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc [sys.catalog]: This master's current role is: LEADER
W20260924 03:01:13.111760 17933 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:01:13.111872 17933 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:01:13.111965 17924 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:13.112129 17924 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:13.113549 17924 catalog_manager.cc:1347] Generated new cluster ID: 28048141fb3040f49b7ceadd37350914
I20260924 03:01:13.113593 17924 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:13.115060 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.109554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37502 (local address 127.0.98.62:37601)
0924 03:01:13.109821 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:13.109829 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:13.109858 (+ 29us) server_negotiation.cc:407] Connection header received
0924 03:01:13.109970 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:13.109977 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:13.110048 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:13.110181 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:13.110425 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.111956 (+ 1531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.112832 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.113409 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.113527 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.114371 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:13.114374 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:13.114375 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:13.114600 (+ 225us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:13.114641 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:13.114656 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:13.114715 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:13.114799 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 03:01:13.114871 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":192,"thread_start_us":95,"threads_started":1}
I20260924 03:01:13.136708 17924 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:13.137220 17924 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:13.141875 17924 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc: Generated new TSK 0
I20260924 03:01:13.142066 17924 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:13.144066 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37601
--builtin_ntp_servers=127.0.98.20:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:01:13.222885 17937 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:13.223078 17937 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:13.223099 17937 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:13.224241 17937 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:13.224282 17937 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:13.224308 17937 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:01:13.225641 17937 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17937
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:13.225868 17937 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:13.226154 17937 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:13.230970 17942 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:13.230983 17943 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:13.231084 17945 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:13.231423 17937 server_base.cc:1034] running on GCE node
I20260924 03:01:13.231673 17937 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:13.231889 17937 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:13.233029 17937 hybrid_clock.cc:648] HybridClock initialized: now 1790218873233013 us; error 30 us; skew 500 ppm
I20260924 03:01:13.234232 17937 webserver.cc:460] Webserver started at http://127.0.98.1:45989/ using document root <none> and password file <none>
I20260924 03:01:13.234472 17937 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:13.234547 17937 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:13.234684 17937 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:13.235792 17937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "e2d491ecdede467a82510eef61ae5f93"
format_stamp: "Formatted at 2026-09-24 03:01:13 on dist-test-slave-sb6p"
server_key: "edd4b1747636ac6e603848933bb8eb1a"
server_key_iv: "822120977ea119caa3e63c06582294cf"
server_key_version: "encryptionkey@0"
I20260924 03:01:13.236151 17937 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "e2d491ecdede467a82510eef61ae5f93"
format_stamp: "Formatted at 2026-09-24 03:01:13 on dist-test-slave-sb6p"
server_key: "edd4b1747636ac6e603848933bb8eb1a"
server_key_iv: "822120977ea119caa3e63c06582294cf"
server_key_version: "encryptionkey@0"
I20260924 03:01:13.237416 17937 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:01:13.238118 17951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:13.238257 17937 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:01:13.238366 17937 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "e2d491ecdede467a82510eef61ae5f93"
format_stamp: "Formatted at 2026-09-24 03:01:13 on dist-test-slave-sb6p"
server_key: "edd4b1747636ac6e603848933bb8eb1a"
server_key_iv: "822120977ea119caa3e63c06582294cf"
server_key_version: "encryptionkey@0"
I20260924 03:01:13.238444 17937 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:13.252633 17937 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:13.252897 17937 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:13.253005 17937 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:13.253187 17937 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:13.253448 17937 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:13.253475 17937 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:13.253502 17937 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:13.253518 17937 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:13.258869 17937 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:39193
I20260924 03:01:13.258924 18064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:39193 every 8 connection(s)
I20260924 03:01:13.259227 17937 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:01:13.268620 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 17937
I20260924 03:01:13.268702 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:01:13.268855 392 external_mini_cluster.cc:1428] Setting key c7fe9b5e5c1c86444a1262b91192c130
I20260924 03:01:13.269716 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.259907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:47715 (local address 127.0.98.62:37601)
0924 03:01:13.259972 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:13.259974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:13.260165 (+ 191us) server_negotiation.cc:407] Connection header received
0924 03:01:13.260732 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:13.260733 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:13.260749 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:13.260800 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:13.261469 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.261967 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.262945 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.263137 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.263191 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.269441 (+ 6250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:13.269442 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:13.269443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:13.269448 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:13.269465 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:13.269470 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:13.269515 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:13.269591 (+ 76us) server_negotiation.cc:299] Negotiation successful
0924 03:01:13.269629 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 03:01:13.269757 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37601
--builtin_ntp_servers=127.0.98.20:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:01:13.269976 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.259873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37601 (local address 127.0.98.1:47715)
0924 03:01:13.260047 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:13.260080 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:13.260615 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:13.260870 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:13.260874 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:13.260884 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:13.261374 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:13.261379 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.262054 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.262055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:13.262853 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:13.262856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.263209 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.263210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:13.263333 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.269250 (+ 5917us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:01:13.269251 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:13.269283 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:01:13.269307 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:13.269522 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:13.269529 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:13.269532 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:13.269596 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 03:01:13.269706 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":74,"threads_started":1}
I20260924 03:01:13.270483 18065 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37601
I20260924 03:01:13.270608 18065 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:13.270790 18065 heartbeater.cc:511] Master 127.0.98.62:37601 requested a full tablet report, sending...
I20260924 03:01:13.271370 17877 ts_manager.cc:194] Registered new tserver with Master: e2d491ecdede467a82510eef61ae5f93 (127.0.98.1:39193)
I20260924 03:01:13.272167 17877 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:47715
W20260924 03:01:13.349063 18068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:13.349258 18068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:13.349279 18068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:13.350420 18068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:13.350459 18068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:13.350490 18068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:01:13.351831 18068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:13.352052 18068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:13.352319 18068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:13.357036 18074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:13.357038 18076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:13.357128 18073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:13.357386 18068 server_base.cc:1034] running on GCE node
I20260924 03:01:13.357573 18068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:13.357784 18068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:13.358911 18068 hybrid_clock.cc:648] HybridClock initialized: now 1790218873358894 us; error 30 us; skew 500 ppm
I20260924 03:01:13.360046 18068 webserver.cc:460] Webserver started at http://127.0.98.2:37571/ using document root <none> and password file <none>
I20260924 03:01:13.360237 18068 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:13.360275 18068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:13.360366 18068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:13.361351 18068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "f09079aa24994ed8a1be05f7ca0de408"
format_stamp: "Formatted at 2026-09-24 03:01:13 on dist-test-slave-sb6p"
server_key: "c3e242b3bdb36535c683d399ebe214fc"
server_key_iv: "31c32a5dec44730e38c37b9094bbdd27"
server_key_version: "encryptionkey@0"
I20260924 03:01:13.361634 18068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "f09079aa24994ed8a1be05f7ca0de408"
format_stamp: "Formatted at 2026-09-24 03:01:13 on dist-test-slave-sb6p"
server_key: "c3e242b3bdb36535c683d399ebe214fc"
server_key_iv: "31c32a5dec44730e38c37b9094bbdd27"
server_key_version: "encryptionkey@0"
I20260924 03:01:13.362792 18068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:01:13.363410 18082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:13.363582 18068 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:13.363655 18068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "f09079aa24994ed8a1be05f7ca0de408"
format_stamp: "Formatted at 2026-09-24 03:01:13 on dist-test-slave-sb6p"
server_key: "c3e242b3bdb36535c683d399ebe214fc"
server_key_iv: "31c32a5dec44730e38c37b9094bbdd27"
server_key_version: "encryptionkey@0"
I20260924 03:01:13.363705 18068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:13.381665 18068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:13.381942 18068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:13.382042 18068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:13.382217 18068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:13.382471 18068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:13.382500 18068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:13.382562 18068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:13.382588 18068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:13.388543 18068 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:42711
I20260924 03:01:13.388610 18195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:42711 every 8 connection(s)
I20260924 03:01:13.389014 18068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:01:13.394956 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 18068
I20260924 03:01:13.395053 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:01:13.395268 392 external_mini_cluster.cc:1428] Setting key e9c8689997994f1feca9f9b3c1c83ed6
I20260924 03:01:13.395918 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.390186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36921 (local address 127.0.98.62:37601)
0924 03:01:13.390246 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:13.390248 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:13.390259 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:01:13.390993 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:13.390994 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:13.391005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:13.391058 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:13.391912 (+ 854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.392662 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.393874 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.394287 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.394360 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.395665 (+ 1305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:13.395666 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:13.395667 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:13.395671 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:13.395685 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:13.395689 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:13.395731 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:13.395809 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 03:01:13.395840 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 03:01:13.396224 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37601
--builtin_ntp_servers=127.0.98.20:33489
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:01:13.396270 18198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.389871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37601 (local address 127.0.98.2:36921)
0924 03:01:13.390098 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:13.390146 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:13.390868 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:13.391148 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:13.391153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:13.391163 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:13.391820 (+ 657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:13.391828 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.392756 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.392757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:13.393792 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:13.393795 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.394365 (+ 570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.394366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:13.394548 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.395412 (+ 864us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:01:13.395413 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:13.395452 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:01:13.395480 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:13.395743 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:13.395748 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:13.395751 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:13.395813 (+ 62us) client_negotiation.cc:240] Negotiation successful
0924 03:01:13.395919 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":87,"threads_started":1}
I20260924 03:01:13.397827 18196 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37601
I20260924 03:01:13.397919 18196 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:13.398098 18196 heartbeater.cc:511] Master 127.0.98.62:37601 requested a full tablet report, sending...
I20260924 03:01:13.398444 17877 ts_manager.cc:194] Registered new tserver with Master: f09079aa24994ed8a1be05f7ca0de408 (127.0.98.2:42711)
I20260924 03:01:13.398947 17877 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:36921
W20260924 03:01:13.486445 18199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:13.486704 18199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:13.486727 18199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:13.487902 18199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:13.487938 18199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:13.487969 18199 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:01:13.489281 18199 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33489
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:37601
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18199
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:13.489481 18199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:13.489738 18199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:13.494642 18204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:13.494684 18199 server_base.cc:1034] running on GCE node
W20260924 03:01:13.494650 18205 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:13.494654 18207 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:13.495051 18199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:13.495285 18199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:13.496436 18199 hybrid_clock.cc:648] HybridClock initialized: now 1790218873496428 us; error 34 us; skew 500 ppm
I20260924 03:01:13.497594 18199 webserver.cc:460] Webserver started at http://127.0.98.3:36313/ using document root <none> and password file <none>
I20260924 03:01:13.497809 18199 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:13.497856 18199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:13.498016 18199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:13.499114 18199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "c8a0867b49e843829692d9b76a7cb8c3"
format_stamp: "Formatted at 2026-09-24 03:01:13 on dist-test-slave-sb6p"
server_key: "f4172be0404348ebb9ed4fc341a70a70"
server_key_iv: "5f6c197dae0239f8124c69f7ca342d37"
server_key_version: "encryptionkey@0"
I20260924 03:01:13.499426 18199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "c8a0867b49e843829692d9b76a7cb8c3"
format_stamp: "Formatted at 2026-09-24 03:01:13 on dist-test-slave-sb6p"
server_key: "f4172be0404348ebb9ed4fc341a70a70"
server_key_iv: "5f6c197dae0239f8124c69f7ca342d37"
server_key_version: "encryptionkey@0"
I20260924 03:01:13.500665 18199 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:01:13.501358 18213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:13.501530 18199 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:13.501610 18199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "c8a0867b49e843829692d9b76a7cb8c3"
format_stamp: "Formatted at 2026-09-24 03:01:13 on dist-test-slave-sb6p"
server_key: "f4172be0404348ebb9ed4fc341a70a70"
server_key_iv: "5f6c197dae0239f8124c69f7ca342d37"
server_key_version: "encryptionkey@0"
I20260924 03:01:13.501672 18199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:13.512491 18199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:13.512792 18199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:13.512919 18199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:13.513123 18199 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:13.513442 18199 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:13.513477 18199 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:13.513506 18199 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:13.513530 18199 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:13.518985 18199 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:40945
I20260924 03:01:13.519037 18326 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:40945 every 8 connection(s)
I20260924 03:01:13.519354 18199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:01:13.521248 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 18199
I20260924 03:01:13.521338 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:01:13.521531 392 external_mini_cluster.cc:1428] Setting key de3d01ca6a6962c193c765e96b8d205a
I20260924 03:01:13.524514 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.520038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:59545 (local address 127.0.98.62:37601)
0924 03:01:13.520118 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:13.520120 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:13.520324 (+ 204us) server_negotiation.cc:407] Connection header received
0924 03:01:13.520812 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:13.520814 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:13.520824 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:13.520855 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:13.521549 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.522236 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.523315 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.523504 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.523554 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.524290 (+ 736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:13.524290 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:13.524291 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:13.524295 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:13.524309 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:13.524313 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:13.524349 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:13.524430 (+ 81us) server_negotiation.cc:299] Negotiation successful
0924 03:01:13.524457 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260924 03:01:13.524700 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.520012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:37601 (local address 127.0.98.3:59545)
0924 03:01:13.520206 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:13.520240 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:13.520722 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:13.520911 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:13.520921 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:13.520930 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:13.521462 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:13.521468 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.522330 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.522331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:13.523239 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:13.523241 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.523592 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.523593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:13.523722 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.524173 (+ 451us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:01:13.524173 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:13.524201 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:01:13.524224 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:13.524373 (+ 149us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:13.524376 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:13.524378 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:01:13.524427 (+ 49us) client_negotiation.cc:240] Negotiation successful
0924 03:01:13.524507 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":87,"threads_started":1}
I20260924 03:01:13.525223 18327 heartbeater.cc:347] Connected to a master server at 127.0.98.62:37601
I20260924 03:01:13.525313 18327 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:13.525490 18327 heartbeater.cc:511] Master 127.0.98.62:37601 requested a full tablet report, sending...
I20260924 03:01:13.525820 17877 ts_manager.cc:194] Registered new tserver with Master: c8a0867b49e843829692d9b76a7cb8c3 (127.0.98.3:40945)
I20260924 03:01:13.526176 17877 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:59545
I20260924 03:01:13.533309 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:01:13.537441 17918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.534669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37518 (local address 127.0.98.62:37601)
0924 03:01:13.534744 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:13.534746 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:13.534820 (+ 74us) server_negotiation.cc:407] Connection header received
0924 03:01:13.534898 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:13.534898 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:13.534910 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:13.534957 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:13.535117 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.535608 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.536374 (+ 766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.536565 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.536618 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.537209 (+ 591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:13.537211 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:13.537211 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:13.537216 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:13.537232 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:13.537235 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:13.537277 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:13.537332 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 03:01:13.537358 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260924 03:01:13.539074 17877 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:01:13.540581 17877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:01:13.546761 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.542774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56406 (local address 127.0.98.1:39193)
0924 03:01:13.543088 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:13.543092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:13.543103 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:01:13.543225 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:13.543226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:13.543240 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:13.543316 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:13.543541 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.544674 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.545843 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.546337 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.546403 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.546500 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:13.546633 (+ 133us) server_negotiation.cc:299] Negotiation successful
0924 03:01:13.546671 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":281,"thread_start_us":102,"threads_started":1}
I20260924 03:01:13.546779 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.542685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39193 (local address 127.0.0.1:56406)
0924 03:01:13.542996 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:13.543036 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:13.543099 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:13.543396 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:13.543397 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:13.543401 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:13.543465 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:13.543467 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.544761 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.544762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:13.545772 (+ 1010us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:13.545774 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.546415 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.546417 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:13.546507 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.546546 (+ 39us) client_negotiation.cc:769] Sending connection context
0924 03:01:13.546617 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 03:01:13.546671 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"mutex_wait_us":4,"thread_start_us":61,"threads_started":1}
I20260924 03:01:13.546949 18338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.542811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55032 (local address 127.0.98.2:42711)
0924 03:01:13.543011 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:13.543016 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:13.543173 (+ 157us) server_negotiation.cc:407] Connection header received
0924 03:01:13.543226 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:13.543228 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:13.543247 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:13.543313 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:13.543536 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.544276 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.545615 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.546346 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.546408 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.546502 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:13.546817 (+ 315us) server_negotiation.cc:299] Negotiation successful
0924 03:01:13.546855 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":77,"threads_started":1}
I20260924 03:01:13.547111 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.542707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42711 (local address 127.0.0.1:55032)
0924 03:01:13.543000 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:13.543036 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:13.543099 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:13.543391 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:13.543394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:13.543396 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:13.543462 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:13.543464 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.544352 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.544353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:13.545524 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:13.545527 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.546554 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.546556 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:13.546618 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.546622 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:01:13.546665 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 03:01:13.546706 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"thread_start_us":83,"threads_started":1}
I20260924 03:01:13.547524 17999 tablet_service.cc:1467] Processing CreateTablet for tablet 80926c3540a84bec8c1ad97bb0a81c23 (DEFAULT_TABLE table=test-table [id=4b750fff67b04ef499855a7d391f66a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:13.547681 18122 tablet_service.cc:1467] Processing CreateTablet for tablet 80926c3540a84bec8c1ad97bb0a81c23 (DEFAULT_TABLE table=test-table [id=4b750fff67b04ef499855a7d391f66a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:13.547891 17999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80926c3540a84bec8c1ad97bb0a81c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:13.548019 18122 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80926c3540a84bec8c1ad97bb0a81c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:13.548584 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.542888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40945 (local address 127.0.0.1:53116)
0924 03:01:13.543225 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:13.543233 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:13.543278 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:13.543495 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:13.543496 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:13.543498 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:13.543533 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:13.543534 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.545854 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.545855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:13.547301 (+ 1446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:13.547304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.548362 (+ 1058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.548363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:13.548422 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.548427 (+ 5us) client_negotiation.cc:769] Sending connection context
0924 03:01:13.548479 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 03:01:13.548518 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":70,"threads_started":1}
I20260924 03:01:13.548610 18340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.542996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53116 (local address 127.0.98.3:40945)
0924 03:01:13.543166 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:13.543170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:13.543285 (+ 115us) server_negotiation.cc:407] Connection header received
0924 03:01:13.543341 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:13.543343 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:13.543357 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:13.543421 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:13.543602 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.544668 (+ 1066us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.547383 (+ 2715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.548260 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.548350 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.548434 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:13.548491 (+ 57us) server_negotiation.cc:299] Negotiation successful
0924 03:01:13.548527 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":77,"threads_started":1}
I20260924 03:01:13.549465 18261 tablet_service.cc:1467] Processing CreateTablet for tablet 80926c3540a84bec8c1ad97bb0a81c23 (DEFAULT_TABLE table=test-table [id=4b750fff67b04ef499855a7d391f66a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:13.549861 18261 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 80926c3540a84bec8c1ad97bb0a81c23. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:13.550760 18341 tablet_bootstrap.cc:492] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93: Bootstrap starting.
I20260924 03:01:13.551227 18342 tablet_bootstrap.cc:492] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408: Bootstrap starting.
I20260924 03:01:13.551425 18341 tablet_bootstrap.cc:654] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:13.551685 18341 log.cc:824] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:13.551846 18342 tablet_bootstrap.cc:654] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:13.552160 18342 log.cc:824] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:13.552397 18341 tablet_bootstrap.cc:492] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93: No bootstrap required, opened a new log
I20260924 03:01:13.552467 18341 ts_tablet_manager.cc:1397] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:01:13.552866 18342 tablet_bootstrap.cc:492] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408: No bootstrap required, opened a new log
I20260924 03:01:13.552969 18342 ts_tablet_manager.cc:1397] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:01:13.553443 18345 tablet_bootstrap.cc:492] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3: Bootstrap starting.
I20260924 03:01:13.554055 18345 tablet_bootstrap.cc:654] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:13.554425 18345 log.cc:824] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:13.554733 18341 raft_consensus.cc:348] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d491ecdede467a82510eef61ae5f93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39193 } } peers { permanent_uuid: "f09079aa24994ed8a1be05f7ca0de408" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42711 } } peers { permanent_uuid: "c8a0867b49e843829692d9b76a7cb8c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 03:01:13.554858 18341 raft_consensus.cc:374] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:13.554888 18341 raft_consensus.cc:729] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e2d491ecdede467a82510eef61ae5f93, State: Initialized, Role: FOLLOWER
I20260924 03:01:13.555011 18341 consensus_queue.cc:260] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d491ecdede467a82510eef61ae5f93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39193 } } peers { permanent_uuid: "f09079aa24994ed8a1be05f7ca0de408" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42711 } } peers { permanent_uuid: "c8a0867b49e843829692d9b76a7cb8c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 03:01:13.555269 18345 tablet_bootstrap.cc:492] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3: No bootstrap required, opened a new log
I20260924 03:01:13.555269 18341 ts_tablet_manager.cc:1428] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:01:13.555332 18345 ts_tablet_manager.cc:1397] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:01:13.555387 18065 heartbeater.cc:503] Master 127.0.98.62:37601 was elected leader, sending a full tablet report...
I20260924 03:01:13.555357 18342 raft_consensus.cc:348] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d491ecdede467a82510eef61ae5f93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39193 } } peers { permanent_uuid: "f09079aa24994ed8a1be05f7ca0de408" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42711 } } peers { permanent_uuid: "c8a0867b49e843829692d9b76a7cb8c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 03:01:13.555517 18342 raft_consensus.cc:374] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:13.555555 18342 raft_consensus.cc:729] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f09079aa24994ed8a1be05f7ca0de408, State: Initialized, Role: FOLLOWER
I20260924 03:01:13.555665 18342 consensus_queue.cc:260] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d491ecdede467a82510eef61ae5f93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39193 } } peers { permanent_uuid: "f09079aa24994ed8a1be05f7ca0de408" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42711 } } peers { permanent_uuid: "c8a0867b49e843829692d9b76a7cb8c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 03:01:13.555895 18342 ts_tablet_manager.cc:1428] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:01:13.555990 18196 heartbeater.cc:503] Master 127.0.98.62:37601 was elected leader, sending a full tablet report...
I20260924 03:01:13.557420 18345 raft_consensus.cc:348] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d491ecdede467a82510eef61ae5f93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39193 } } peers { permanent_uuid: "f09079aa24994ed8a1be05f7ca0de408" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42711 } } peers { permanent_uuid: "c8a0867b49e843829692d9b76a7cb8c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 03:01:13.557543 18345 raft_consensus.cc:374] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:13.557564 18345 raft_consensus.cc:729] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8a0867b49e843829692d9b76a7cb8c3, State: Initialized, Role: FOLLOWER
I20260924 03:01:13.557647 18345 consensus_queue.cc:260] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d491ecdede467a82510eef61ae5f93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39193 } } peers { permanent_uuid: "f09079aa24994ed8a1be05f7ca0de408" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42711 } } peers { permanent_uuid: "c8a0867b49e843829692d9b76a7cb8c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 03:01:13.557842 18345 ts_tablet_manager.cc:1428] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:01:13.557897 18327 heartbeater.cc:503] Master 127.0.98.62:37601 was elected leader, sending a full tablet report...
W20260924 03:01:13.639617 18197 tablet.cc:2369] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:01:13.758764 18347 raft_consensus.cc:482] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:01:13.758888 18347 raft_consensus.cc:504] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d491ecdede467a82510eef61ae5f93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39193 } } peers { permanent_uuid: "f09079aa24994ed8a1be05f7ca0de408" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42711 } } peers { permanent_uuid: "c8a0867b49e843829692d9b76a7cb8c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 03:01:13.759193 18347 leader_election.cc:288] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f09079aa24994ed8a1be05f7ca0de408 (127.0.98.2:42711), c8a0867b49e843829692d9b76a7cb8c3 (127.0.98.3:40945)
W20260924 03:01:13.759863 18066 tablet.cc:2369] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:01:13.762861 18338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.759429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:42669 (local address 127.0.98.2:42711)
0924 03:01:13.759490 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:13.759492 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:13.759502 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:01:13.759592 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:13.759593 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:13.759603 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:13.759650 (+ 47us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:13.759797 (+ 147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.760567 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.761956 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.762555 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.762608 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.762616 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:13.762755 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 03:01:13.762790 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 03:01:13.762861 18340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.759557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:51201 (local address 127.0.98.3:40945)
0924 03:01:13.759608 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:13.759610 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:13.759622 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:01:13.759707 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:13.759708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:13.759737 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:13.759779 (+ 42us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:13.759967 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.760627 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.761953 (+ 1326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.762505 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.762602 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.762612 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:13.762756 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 03:01:13.762790 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:01:13.762892 18067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.759365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:42711 (local address 127.0.98.1:42669)
0924 03:01:13.759421 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:13.759434 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:13.759498 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:13.759655 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:13.759656 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:13.759657 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:13.759724 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:13.759726 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.760633 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.760633 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:13.761859 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:13.761862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.762607 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.762608 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:13.762684 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.762690 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:01:13.762747 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 03:01:13.762796 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":18}
I20260924 03:01:13.762897 18350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.759461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:40945 (local address 127.0.98.1:51201)
0924 03:01:13.759574 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:13.759585 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:13.759640 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:13.759826 (+ 186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:13.759827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:13.759828 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:13.759886 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:13.759888 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.760691 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.760692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:13.761863 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:13.761864 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.762602 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.762603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:13.762684 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.762690 (+ 6us) client_negotiation.cc:769] Sending connection context
0924 03:01:13.762747 (+ 57us) client_negotiation.cc:240] Negotiation successful
0924 03:01:13.762803 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":35,"thread_start_us":64,"threads_started":1}
I20260924 03:01:13.763226 18148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80926c3540a84bec8c1ad97bb0a81c23" candidate_uuid: "e2d491ecdede467a82510eef61ae5f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f09079aa24994ed8a1be05f7ca0de408" is_pre_election: true
I20260924 03:01:13.763389 18148 raft_consensus.cc:2454] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2d491ecdede467a82510eef61ae5f93 in term 0.
I20260924 03:01:13.763360 18281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80926c3540a84bec8c1ad97bb0a81c23" candidate_uuid: "e2d491ecdede467a82510eef61ae5f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a0867b49e843829692d9b76a7cb8c3" is_pre_election: true
I20260924 03:01:13.763491 18281 raft_consensus.cc:2454] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e2d491ecdede467a82510eef61ae5f93 in term 0.
I20260924 03:01:13.763602 17953 leader_election.cc:302] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2d491ecdede467a82510eef61ae5f93, f09079aa24994ed8a1be05f7ca0de408; no voters:
I20260924 03:01:13.763731 18347 raft_consensus.cc:2781] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:01:13.763792 18347 raft_consensus.cc:482] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:01:13.763815 18347 raft_consensus.cc:3037] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:13.764529 18347 raft_consensus.cc:504] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d491ecdede467a82510eef61ae5f93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39193 } } peers { permanent_uuid: "f09079aa24994ed8a1be05f7ca0de408" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42711 } } peers { permanent_uuid: "c8a0867b49e843829692d9b76a7cb8c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 03:01:13.764643 18347 leader_election.cc:288] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [CANDIDATE]: Term 1 election: Requested vote from peers f09079aa24994ed8a1be05f7ca0de408 (127.0.98.2:42711), c8a0867b49e843829692d9b76a7cb8c3 (127.0.98.3:40945)
I20260924 03:01:13.764824 18281 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80926c3540a84bec8c1ad97bb0a81c23" candidate_uuid: "e2d491ecdede467a82510eef61ae5f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c8a0867b49e843829692d9b76a7cb8c3"
I20260924 03:01:13.764827 18148 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "80926c3540a84bec8c1ad97bb0a81c23" candidate_uuid: "e2d491ecdede467a82510eef61ae5f93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f09079aa24994ed8a1be05f7ca0de408"
I20260924 03:01:13.764904 18281 raft_consensus.cc:3037] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:13.764904 18148 raft_consensus.cc:3037] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:13.765625 18148 raft_consensus.cc:2454] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2d491ecdede467a82510eef61ae5f93 in term 1.
I20260924 03:01:13.765674 18281 raft_consensus.cc:2454] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e2d491ecdede467a82510eef61ae5f93 in term 1.
I20260924 03:01:13.765780 17953 leader_election.cc:302] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e2d491ecdede467a82510eef61ae5f93, f09079aa24994ed8a1be05f7ca0de408; no voters:
I20260924 03:01:13.765867 18347 raft_consensus.cc:2781] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:13.766017 18347 raft_consensus.cc:686] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [term 1 LEADER]: Becoming Leader. State: Replica: e2d491ecdede467a82510eef61ae5f93, State: Running, Role: LEADER
I20260924 03:01:13.766099 18347 consensus_queue.cc:237] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d491ecdede467a82510eef61ae5f93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39193 } } peers { permanent_uuid: "f09079aa24994ed8a1be05f7ca0de408" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42711 } } peers { permanent_uuid: "c8a0867b49e843829692d9b76a7cb8c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } }
I20260924 03:01:13.766909 17877 catalog_manager.cc:5495] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 reported cstate change: term changed from 0 to 1, leader changed from <none> to e2d491ecdede467a82510eef61ae5f93 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "e2d491ecdede467a82510eef61ae5f93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e2d491ecdede467a82510eef61ae5f93" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 39193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f09079aa24994ed8a1be05f7ca0de408" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42711 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c8a0867b49e843829692d9b76a7cb8c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 } health_report { overall_health: UNKNOWN } } }
W20260924 03:01:13.769843 18328 tablet.cc:2369] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:01:13.771595 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.769384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.0.98.1:39193)
0924 03:01:13.769436 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:13.769438 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:13.769447 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:01:13.769500 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:13.769501 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:13.769516 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:13.769568 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:01:13.769642 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.770280 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.771003 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:13.771197 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:13.771241 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:13.771344 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:01:13.771435 (+ 91us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:01:13.771488 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:13.771515 (+ 27us) server_negotiation.cc:299] Negotiation successful
0924 03:01:13.771530 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 03:01:13.773072 18148 raft_consensus.cc:1261] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408 [term 1 FOLLOWER]: Refusing update from remote peer e2d491ecdede467a82510eef61ae5f93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:01:13.773069 18281 raft_consensus.cc:1261] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3 [term 1 FOLLOWER]: Refusing update from remote peer e2d491ecdede467a82510eef61ae5f93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:01:13.773403 18351 consensus_queue.cc:1035] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c8a0867b49e843829692d9b76a7cb8c3" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 40945 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:01:13.773518 18347 consensus_queue.cc:1035] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f09079aa24994ed8a1be05f7ca0de408" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 42711 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:01:13.775799 18356 mvcc.cc:205] Tried to move back new op lower bound from 7332736506972282880 to 7332736506946588672. Current Snapshot: MvccSnapshot[applied={T|T < 7332736506972282880}]
I20260924 03:01:13.777387 18355 mvcc.cc:205] Tried to move back new op lower bound from 7332736506972282880 to 7332736506946588672. Current Snapshot: MvccSnapshot[applied={T|T < 7332736506972282880 or (T in {7332736506972282880})}]
I20260924 03:01:13.777693 17877 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:01:13.777770 17877 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:01:13.778447 17877 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cadc472948db491c8b089183f978a3cc: Sending DeleteTablet for 3 replicas of tablet 80926c3540a84bec8c1ad97bb0a81c23
I20260924 03:01:13.778780 18122 tablet_service.cc:1514] Processing DeleteTablet for tablet 80926c3540a84bec8c1ad97bb0a81c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:13 UTC) from {username='slave'} at 127.0.0.1:55032
I20260924 03:01:13.778823 17999 tablet_service.cc:1514] Processing DeleteTablet for tablet 80926c3540a84bec8c1ad97bb0a81c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:13 UTC) from {username='slave'} at 127.0.0.1:56406
I20260924 03:01:13.779045 18364 tablet_replica.cc:330] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408: stopping tablet replica
I20260924 03:01:13.779088 18365 tablet_replica.cc:330] T 80926c3540a84bec8c1ad97bb0a81c23 P e2d491ecdede467a82510eef61ae5f93: stopping tablet replica
I20260924 03:01:13.779114 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 17937
I20260924 03:01:13.779244 18364 raft_consensus.cc:2229] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:01:13.779853 18364 raft_consensus.cc:2258] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:01:13.780252 18364 ts_tablet_manager.cc:1905] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:01:13.781824 18364 ts_tablet_manager.cc:1918] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:01:13.781888 18364 log.cc:1196] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/80926c3540a84bec8c1ad97bb0a81c23
I20260924 03:01:13.782099 18364 ts_tablet_manager.cc:1939] T 80926c3540a84bec8c1ad97bb0a81c23 P f09079aa24994ed8a1be05f7ca0de408: Deleting consensus metadata
I20260924 03:01:13.782456 17862 catalog_manager.cc:4841] TS f09079aa24994ed8a1be05f7ca0de408 (127.0.98.2:42711): tablet 80926c3540a84bec8c1ad97bb0a81c23 (table test-table [id=4b750fff67b04ef499855a7d391f66a0]) successfully deleted
I20260924 03:01:13.783394 18261 tablet_service.cc:1514] Processing DeleteTablet for tablet 80926c3540a84bec8c1ad97bb0a81c23 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:13 UTC) from {username='slave'} at 127.0.0.1:53116
I20260924 03:01:13.783617 18366 tablet_replica.cc:330] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3: stopping tablet replica
I20260924 03:01:13.783711 18366 raft_consensus.cc:2229] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:01:13.783820 18366 raft_consensus.cc:2258] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260924 03:01:13.784101 17861 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:39193 (error 108)
I20260924 03:01:13.784376 18366 ts_tablet_manager.cc:1905] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:01:13.784584 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:13.784396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:39193 (local address 127.0.0.1:52956)
0924 03:01:13.784442 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:13.784508 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 03:01:13.785239 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 18068
I20260924 03:01:13.785907 18366 ts_tablet_manager.cc:1918] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:01:13.785989 18366 log.cc:1196] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/80926c3540a84bec8c1ad97bb0a81c23
W20260924 03:01:13.785993 17861 catalog_manager.cc:4584] TS e2d491ecdede467a82510eef61ae5f93 (127.0.98.1:39193): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 80926c3540a84bec8c1ad97bb0a81c23: Network error: Client connection negotiation failed: client connection to 127.0.98.1:39193: connect: Connection refused (error 111)
I20260924 03:01:13.786222 18366 ts_tablet_manager.cc:1939] T 80926c3540a84bec8c1ad97bb0a81c23 P c8a0867b49e843829692d9b76a7cb8c3: Deleting consensus metadata
I20260924 03:01:13.786662 17862 catalog_manager.cc:4841] TS c8a0867b49e843829692d9b76a7cb8c3 (127.0.98.3:40945): tablet 80926c3540a84bec8c1ad97bb0a81c23 (table test-table [id=4b750fff67b04ef499855a7d391f66a0]) successfully deleted
I20260924 03:01:13.790891 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 18199
I20260924 03:01:13.796710 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 17846
2026-09-24T03:01:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (825 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:13 dist-test-slave-sb6p krb5kdc[18370](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:13 dist-test-slave-sb6p krb5kdc[18370](info): set up 2 sockets
Sep 24 03:01:13 dist-test-slave-sb6p krb5kdc[18370](info): commencing operation
krb5kdc: starting...
W20260924 03:01:14.734179 392 mini_kdc.cc:121] Time spent starting KDC: real 0.919s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:14 dist-test-slave-sb6p krb5kdc[18370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:01:14.844554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44095
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44095
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:14.923548 18383 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:14.923748 18383 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:14.923777 18383 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:14.924922 18383 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:14.924968 18383 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:14.924988 18383 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:14.925004 18383 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:14.925027 18383 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:14.926476 18383 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44095
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18383
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:14.926729 18383 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:14.926932 18383 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:14.931406 18389 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:14.931413 18388 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:14.931623 18383 server_base.cc:1034] running on GCE node
W20260924 03:01:14.931499 18391 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:14.931881 18383 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:14.932132 18383 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:14.933283 18383 hybrid_clock.cc:648] HybridClock initialized: now 1790218874933253 us; error 49 us; skew 500 ppm
Sep 24 03:01:14 dist-test-slave-sb6p krb5kdc[18370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:14.935600 18383 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:01:14.936024 18383 webserver.cc:460] Webserver started at http://127.0.98.62:42735/ using document root <none> and password file <none>
I20260924 03:01:14.936213 18383 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:14.936250 18383 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:14.936339 18383 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:14.937327 18383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "d44ef2dac6be44e3a0fa9bb5ad7b382c"
format_stamp: "Formatted at 2026-09-24 03:01:14 on dist-test-slave-sb6p"
server_key: "cf930e3b87f7bcb2b39a0b51057b6ffe"
server_key_iv: "cf4cb7e8ad18cd7d2efc550e13556240"
server_key_version: "encryptionkey@0"
I20260924 03:01:14.937608 18383 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "d44ef2dac6be44e3a0fa9bb5ad7b382c"
format_stamp: "Formatted at 2026-09-24 03:01:14 on dist-test-slave-sb6p"
server_key: "cf930e3b87f7bcb2b39a0b51057b6ffe"
server_key_iv: "cf4cb7e8ad18cd7d2efc550e13556240"
server_key_version: "encryptionkey@0"
I20260924 03:01:14.938760 18383 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:01:14.939427 18398 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:14.939616 18383 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:14.939694 18383 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "d44ef2dac6be44e3a0fa9bb5ad7b382c"
format_stamp: "Formatted at 2026-09-24 03:01:14 on dist-test-slave-sb6p"
server_key: "cf930e3b87f7bcb2b39a0b51057b6ffe"
server_key_iv: "cf4cb7e8ad18cd7d2efc550e13556240"
server_key_version: "encryptionkey@0"
I20260924 03:01:14.939749 18383 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:14.961977 18383 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:14.964077 18383 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:14.964188 18383 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:14.967743 18383 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44095
I20260924 03:01:14.967792 18450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44095 every 8 connection(s)
I20260924 03:01:14.968102 18383 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:14.968904 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 18383
I20260924 03:01:14.968935 18451 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:14.968997 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:14.969184 392 external_mini_cluster.cc:1428] Setting key e5b92411addd969899b0217b2f5145d4
I20260924 03:01:14.971935 18451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c: Bootstrap starting.
I20260924 03:01:14.972534 18451 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:14.972853 18451 log.cc:824] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:14.973510 18451 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c: No bootstrap required, opened a new log
Sep 24 03:01:14 dist-test-slave-sb6p krb5kdc[18370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:14.975770 18451 raft_consensus.cc:348] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d44ef2dac6be44e3a0fa9bb5ad7b382c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } }
I20260924 03:01:14.975919 18451 raft_consensus.cc:374] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:14.975958 18451 raft_consensus.cc:729] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d44ef2dac6be44e3a0fa9bb5ad7b382c, State: Initialized, Role: FOLLOWER
I20260924 03:01:14.976089 18451 consensus_queue.cc:260] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d44ef2dac6be44e3a0fa9bb5ad7b382c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } }
I20260924 03:01:14.976169 18451 raft_consensus.cc:388] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:14.976208 18451 raft_consensus.cc:482] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:14.976255 18451 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:14.976756 18451 raft_consensus.cc:504] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d44ef2dac6be44e3a0fa9bb5ad7b382c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } }
I20260924 03:01:14.976838 18451 leader_election.cc:302] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d44ef2dac6be44e3a0fa9bb5ad7b382c; no voters:
I20260924 03:01:14.976976 18451 leader_election.cc:288] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:14.977029 18456 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:14.977180 18456 raft_consensus.cc:686] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [term 1 LEADER]: Becoming Leader. State: Replica: d44ef2dac6be44e3a0fa9bb5ad7b382c, State: Running, Role: LEADER
I20260924 03:01:14.977240 18451 sys_catalog.cc:564] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:14.977334 18456 consensus_queue.cc:237] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d44ef2dac6be44e3a0fa9bb5ad7b382c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } }
I20260924 03:01:14.978261 18457 sys_catalog.cc:455] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d44ef2dac6be44e3a0fa9bb5ad7b382c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d44ef2dac6be44e3a0fa9bb5ad7b382c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } } }
I20260924 03:01:14.978350 18457 sys_catalog.cc:458] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:14.978431 18458 sys_catalog.cc:455] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [sys.catalog]: SysCatalogTable state changed. Reason: New leader d44ef2dac6be44e3a0fa9bb5ad7b382c. Latest consensus state: current_term: 1 leader_uuid: "d44ef2dac6be44e3a0fa9bb5ad7b382c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d44ef2dac6be44e3a0fa9bb5ad7b382c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44095 } } }
I20260924 03:01:14.978690 18458 sys_catalog.cc:458] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c [sys.catalog]: This master's current role is: LEADER
W20260924 03:01:14.978981 18471 catalog_manager.cc:1558] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:01:14.979034 18471 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:01:14.979140 18472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:14.979293 18472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:14.980262 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:14.969668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36974 (local address 127.0.98.62:44095)
0924 03:01:14.969804 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:14.969812 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:14.969836 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:01:14.970300 (+ 464us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:14.970306 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:14.970354 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:14.970475 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:14.971278 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:14.972282 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:14.973270 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:14.973754 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:14.973838 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:14.976326 (+ 2488us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:14.976330 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:14.976331 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:14.976343 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:14.978277 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:14.978797 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:14.978800 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:14.978801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:14.978905 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:14.979120 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:14.979120 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:14.979121 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:14.979435 (+ 314us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:14.979561 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:14.980000 (+ 439us) server_negotiation.cc:299] Negotiation successful
0924 03:01:14.980108 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":54,"threads_started":1}
I20260924 03:01:14.980883 18472 catalog_manager.cc:1347] Generated new cluster ID: 6c7280e6d14949449164f5999e688680
I20260924 03:01:14.980937 18472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:14.998726 18472 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:14.999246 18472 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:15.008438 18472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c: Generated new TSK 0
I20260924 03:01:15.008643 18472 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:01:15.035003 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44095
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:15.136654 18477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:15.136847 18477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:15.136868 18477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:15.138017 18477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:15.138057 18477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:15.138082 18477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:01:15.139557 18477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:15.139730 18477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:15.139906 18477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:15.144655 18483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:15.144680 18482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:15.144719 18485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:15.144858 18477 server_base.cc:1034] running on GCE node
I20260924 03:01:15.145056 18477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:15.145268 18477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:15.146440 18477 hybrid_clock.cc:648] HybridClock initialized: now 1790218875146423 us; error 27 us; skew 500 ppm
Sep 24 03:01:15 dist-test-slave-sb6p krb5kdc[18370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:15.148756 18477 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:01:15.149139 18477 webserver.cc:460] Webserver started at http://127.0.98.1:40873/ using document root <none> and password file <none>
I20260924 03:01:15.149303 18477 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:15.149340 18477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:15.149434 18477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:15.150413 18477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "dd1783752056447ca63dbdcb0181fbf9"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "e06e3a2b47c6a6cb95e3f991aad403b5"
server_key_iv: "25d767fff81117419c895993597bb879"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.150745 18477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "dd1783752056447ca63dbdcb0181fbf9"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "e06e3a2b47c6a6cb95e3f991aad403b5"
server_key_iv: "25d767fff81117419c895993597bb879"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.151914 18477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:01:15.152580 18492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:15.152742 18477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:15.152822 18477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "dd1783752056447ca63dbdcb0181fbf9"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "e06e3a2b47c6a6cb95e3f991aad403b5"
server_key_iv: "25d767fff81117419c895993597bb879"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.152884 18477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:15.164937 18477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:15.166630 18477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:15.166738 18477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:15.166911 18477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:15.167153 18477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:15.167181 18477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:15.167204 18477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:15.167220 18477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:15.172370 18477 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:41071
I20260924 03:01:15.172443 18605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:41071 every 8 connection(s)
I20260924 03:01:15.172709 18477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
Sep 24 03:01:15 dist-test-slave-sb6p krb5kdc[18370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:15.179993 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 18477
I20260924 03:01:15.180094 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:01:15.180308 392 external_mini_cluster.cc:1428] Setting key ca4410016dec8ce1bfc9d3bb80fe299f
I20260924 03:01:15.181567 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.173363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:52977 (local address 127.0.98.62:44095)
0924 03:01:15.173406 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:15.173408 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:15.173624 (+ 216us) server_negotiation.cc:407] Connection header received
0924 03:01:15.174564 (+ 940us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:15.174566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:15.174579 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:15.174625 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:15.175238 (+ 613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.175737 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.176740 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.176956 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.177012 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.179090 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:15.179091 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:15.179092 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:15.179098 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:15.180233 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:15.180630 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:15.180631 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:15.180632 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:15.180688 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:15.180934 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:15.180935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:15.180936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:15.181066 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:15.181132 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:15.181371 (+ 239us) server_negotiation.cc:299] Negotiation successful
0924 03:01:15.181476 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:01:15.181634 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.173350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44095 (local address 127.0.98.1:52977)
0924 03:01:15.173511 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:15.173547 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:15.174443 (+ 896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:15.174706 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:15.174710 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:15.174959 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:15.175151 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.175156 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.175813 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.175814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.176660 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.176662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.177023 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.177024 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.177150 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.177582 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:15.177590 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:15.179015 (+ 1425us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:15.180301 (+ 1286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:15.180308 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:15.180310 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:15.180575 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:15.180741 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:15.180742 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:15.180742 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:15.180866 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:15.181123 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:15.181128 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:15.181249 (+ 121us) client_negotiation.cc:769] Sending connection context
0924 03:01:15.181331 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 03:01:15.181449 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260924 03:01:15.182343 18606 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44095
I20260924 03:01:15.182439 18606 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:15.182634 18606 heartbeater.cc:511] Master 127.0.98.62:44095 requested a full tablet report, sending...
I20260924 03:01:15.183145 18415 ts_manager.cc:194] Registered new tserver with Master: dd1783752056447ca63dbdcb0181fbf9 (127.0.98.1:41071)
I20260924 03:01:15.183987 18415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:52977
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:01:15.202695 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44095
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:15.281311 18611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:15.281525 18611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:15.281549 18611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:15.282714 18611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:15.282754 18611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:15.282779 18611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:01:15.284204 18611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18611
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:15.284384 18611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:15.284569 18611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:01:15.289439 18611 server_base.cc:1034] running on GCE node
W20260924 03:01:15.289404 18616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:15.289402 18617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:15.289382 18619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:15.289805 18611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:15.290032 18611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:15.291163 18611 hybrid_clock.cc:648] HybridClock initialized: now 1790218875291148 us; error 34 us; skew 500 ppm
Sep 24 03:01:15 dist-test-slave-sb6p krb5kdc[18370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:15.293603 18611 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:01:15.294001 18611 webserver.cc:460] Webserver started at http://127.0.98.2:36203/ using document root <none> and password file <none>
I20260924 03:01:15.294185 18611 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:15.294227 18611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:15.294348 18611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:15.295403 18611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "6c1ff8afd8094f7fbb4a7abf84ec678d"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "a033d6f9294d927f865fedd4d6224dfc"
server_key_iv: "304584896ead871cc3d235fc7a7a60fd"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.295714 18611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "6c1ff8afd8094f7fbb4a7abf84ec678d"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "a033d6f9294d927f865fedd4d6224dfc"
server_key_iv: "304584896ead871cc3d235fc7a7a60fd"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.296911 18611 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:01:15.297605 18626 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:15.297765 18611 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:15.297848 18611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "6c1ff8afd8094f7fbb4a7abf84ec678d"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "a033d6f9294d927f865fedd4d6224dfc"
server_key_iv: "304584896ead871cc3d235fc7a7a60fd"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.297905 18611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:15.309686 18611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:15.311520 18611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:15.311626 18611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:15.311815 18611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:15.312055 18611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:15.312083 18611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:15.312104 18611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:15.312120 18611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:15.317358 18611 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:46263
I20260924 03:01:15.317430 18739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:46263 every 8 connection(s)
I20260924 03:01:15.317691 18611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
Sep 24 03:01:15 dist-test-slave-sb6p krb5kdc[18370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:15.325990 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.318348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:36447 (local address 127.0.98.62:44095)
0924 03:01:15.318395 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:15.318397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:15.318634 (+ 237us) server_negotiation.cc:407] Connection header received
0924 03:01:15.319529 (+ 895us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:15.319530 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:15.319541 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:15.319592 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:15.320171 (+ 579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.320704 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.321633 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.321829 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.321878 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.323770 (+ 1892us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:15.323772 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:15.323772 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:15.323778 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:15.325004 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:15.325295 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:15.325296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:15.325296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:15.325340 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:15.325500 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:15.325500 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:15.325501 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:15.325594 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:15.325644 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:15.325857 (+ 213us) server_negotiation.cc:299] Negotiation successful
0924 03:01:15.325923 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:01:15.326148 18742 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.318317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44095 (local address 127.0.98.2:36447)
0924 03:01:15.318485 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:15.318541 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:15.319428 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:15.319646 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:15.319650 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:15.319892 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:15.320091 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.320095 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.320769 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.320770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.321558 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.321559 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.321885 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.321885 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.321983 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.322390 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:15.322397 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:15.323691 (+ 1294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:15.325062 (+ 1371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:15.325065 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:15.325066 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:15.325247 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:15.325380 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:15.325380 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:15.325380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:15.325459 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:15.325645 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:15.325650 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:15.325760 (+ 110us) client_negotiation.cc:769] Sending connection context
0924 03:01:15.325837 (+ 77us) client_negotiation.cc:240] Negotiation successful
0924 03:01:15.325950 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":101,"threads_started":1}
I20260924 03:01:15.326608 18740 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44095
I20260924 03:01:15.326707 18740 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:15.326843 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 18611
I20260924 03:01:15.326912 18740 heartbeater.cc:511] Master 127.0.98.62:44095 requested a full tablet report, sending...
I20260924 03:01:15.326960 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:01:15.327154 392 external_mini_cluster.cc:1428] Setting key 8a19fcd30367b855ac75c7fefc0867d6
I20260924 03:01:15.327292 18415 ts_manager.cc:194] Registered new tserver with Master: 6c1ff8afd8094f7fbb4a7abf84ec678d (127.0.98.2:46263)
I20260924 03:01:15.327903 18415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:36447
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:01:15.349432 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44095
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:15.428051 18745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:15.428256 18745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:15.428284 18745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:15.429461 18745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:15.429502 18745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:15.429528 18745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:01:15.431015 18745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44095
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18745
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:15.431229 18745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:15.431440 18745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:15.436172 18751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:15.436148 18750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:15.436146 18753 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:15.436267 18745 server_base.cc:1034] running on GCE node
I20260924 03:01:15.436569 18745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:15.436787 18745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:15.437927 18745 hybrid_clock.cc:648] HybridClock initialized: now 1790218875437908 us; error 35 us; skew 500 ppm
Sep 24 03:01:15 dist-test-slave-sb6p krb5kdc[18370](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:15.440212 18745 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:01:15.440649 18745 webserver.cc:460] Webserver started at http://127.0.98.3:39167/ using document root <none> and password file <none>
I20260924 03:01:15.440814 18745 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:15.440852 18745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:15.440934 18745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:15.441915 18745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "1d2ce31cc30f4311bb78221838288a07"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "674d28ea97c3ddbf316ed2b4bdd0f1e2"
server_key_iv: "a3a2cec5343262133567bb7823c12946"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.442219 18745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "1d2ce31cc30f4311bb78221838288a07"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "674d28ea97c3ddbf316ed2b4bdd0f1e2"
server_key_iv: "a3a2cec5343262133567bb7823c12946"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.443573 18745 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:01:15.444208 18760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:15.444351 18745 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:01:15.444422 18745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "1d2ce31cc30f4311bb78221838288a07"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "674d28ea97c3ddbf316ed2b4bdd0f1e2"
server_key_iv: "a3a2cec5343262133567bb7823c12946"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.444489 18745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:15.470654 18745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:15.472424 18745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:15.472549 18745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:15.472764 18745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:15.473061 18745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:15.473094 18745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:15.473117 18745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:15.473140 18745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:15.478771 18745 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:35747
I20260924 03:01:15.478829 18873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:35747 every 8 connection(s)
I20260924 03:01:15.479141 18745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:01:15.483819 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 18745
I20260924 03:01:15.483909 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:01:15.484062 392 external_mini_cluster.cc:1428] Setting key 4d6702c0bde9f7951b44f89e97fadbc8
Sep 24 03:01:15 dist-test-slave-sb6p krb5kdc[18370](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:15.488224 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.479856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:55225 (local address 127.0.98.62:44095)
0924 03:01:15.479920 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:15.479922 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:15.480188 (+ 266us) server_negotiation.cc:407] Connection header received
0924 03:01:15.481217 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:15.481218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:15.481228 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:15.481284 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:15.481953 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.482497 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.483522 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.483713 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.483782 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.485952 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:15.485953 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:15.485953 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:15.485961 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:15.487121 (+ 1160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:15.487454 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:15.487454 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:15.487455 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:15.487497 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:15.487706 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:15.487708 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:15.487708 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:15.487828 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:15.487894 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:15.488094 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 03:01:15.488158 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260924 03:01:15.488391 18876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.479840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44095 (local address 127.0.98.3:55225)
0924 03:01:15.480036 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:15.480096 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:15.481099 (+ 1003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:15.481399 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:15.481402 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:15.481653 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:15.481851 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.481856 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.482613 (+ 757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.482614 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.483430 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.483432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.483807 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.483808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.483925 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.484430 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:15.484440 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:15.485848 (+ 1408us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:15.487192 (+ 1344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:15.487198 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:15.487199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:15.487402 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:15.487541 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:15.487541 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:15.487542 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:15.487626 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:15.487887 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:15.487889 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:15.488004 (+ 115us) client_negotiation.cc:769] Sending connection context
0924 03:01:15.488075 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 03:01:15.488191 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":79,"threads_started":1}
I20260924 03:01:15.488814 18874 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44095
I20260924 03:01:15.488905 18874 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:15.489071 18874 heartbeater.cc:511] Master 127.0.98.62:44095 requested a full tablet report, sending...
I20260924 03:01:15.489390 18415 ts_manager.cc:194] Registered new tserver with Master: 1d2ce31cc30f4311bb78221838288a07 (127.0.98.3:35747)
I20260924 03:01:15.489729 18415 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:55225
I20260924 03:01:15.495486 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:01:15.502144 18454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.496692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36980 (local address 127.0.98.62:44095)
0924 03:01:15.496742 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:15.496743 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:15.496816 (+ 73us) server_negotiation.cc:407] Connection header received
0924 03:01:15.496866 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:15.496866 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:15.496873 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:15.496925 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:15.497391 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.497873 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.498608 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.498785 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.498832 (+ 47us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.499926 (+ 1094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:15.499927 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:15.499928 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:15.499934 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:15.501169 (+ 1235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:15.501555 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:15.501556 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:15.501556 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:15.501613 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:15.501764 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:15.501765 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:15.501765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:15.501879 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:15.501952 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:15.502036 (+ 84us) server_negotiation.cc:299] Negotiation successful
0924 03:01:15.502082 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 03:01:15.503599 18415 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36980:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:01:15.505052 18415 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:01:15.511286 18883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.507270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41071 (local address 127.0.0.1:38970)
0924 03:01:15.507361 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:15.507397 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:15.507460 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:15.507714 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:15.507717 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:15.507719 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:15.507794 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.507795 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.509149 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.509150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.510139 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.510141 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.511060 (+ 919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.511061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.511129 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.511140 (+ 11us) client_negotiation.cc:769] Sending connection context
0924 03:01:15.511193 (+ 53us) client_negotiation.cc:240] Negotiation successful
0924 03:01:15.511231 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"thread_start_us":45,"threads_started":1}
I20260924 03:01:15.511322 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.507401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38970 (local address 127.0.98.1:41071)
0924 03:01:15.507581 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:15.507586 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:15.507595 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:01:15.507624 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:15.507626 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:15.507637 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:15.507700 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:15.507868 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.509036 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.510220 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.510985 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.511061 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.511115 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:15.511206 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 03:01:15.511244 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":77,"threads_started":1}
I20260924 03:01:15.511768 18886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.507078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46263 (local address 127.0.0.1:41560)
0924 03:01:15.507355 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:15.507396 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:15.507461 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:15.508104 (+ 643us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:15.508105 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:15.508107 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:15.508163 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.508164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.509374 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.509375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.510650 (+ 1275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.510653 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.511548 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.511549 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.511621 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.511624 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:15.511668 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 03:01:15.511705 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"mutex_wait_us":50,"thread_start_us":62,"threads_started":1}
I20260924 03:01:15.511802 18885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.507191 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41560 (local address 127.0.98.2:46263)
0924 03:01:15.507839 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:15.507843 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:15.507871 (+ 28us) server_negotiation.cc:407] Connection header received
0924 03:01:15.507936 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:15.507938 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:15.507954 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:15.508044 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:15.508276 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.509285 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.510753 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.511460 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.511533 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.511587 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:15.511692 (+ 105us) server_negotiation.cc:299] Negotiation successful
0924 03:01:15.511734 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":71,"threads_started":1}
I20260924 03:01:15.512020 18540 tablet_service.cc:1467] Processing CreateTablet for tablet e71200e6971e473087d26e7280851b60 (DEFAULT_TABLE table=test-table [id=de2dd2dae5d44c88af649bcaf1de5cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:15.512396 18540 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71200e6971e473087d26e7280851b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:15.512692 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.507067 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35747 (local address 127.0.0.1:56674)
0924 03:01:15.507355 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:15.507396 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:15.507461 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:15.508391 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:15.508392 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:15.508394 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:15.508446 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.508447 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.509395 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.509396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.510653 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.510654 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.512468 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.512469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.512537 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.512541 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:01:15.512593 (+ 52us) client_negotiation.cc:240] Negotiation successful
0924 03:01:15.512630 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":72,"threads_started":1}
I20260924 03:01:15.512730 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.507160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56674 (local address 127.0.98.3:35747)
0924 03:01:15.508228 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:15.508235 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:15.508259 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:01:15.508309 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:15.508311 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:15.508326 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:15.508393 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:15.508513 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.509314 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.511622 (+ 2308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.512315 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.512378 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.512432 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:15.512613 (+ 181us) server_negotiation.cc:299] Negotiation successful
0924 03:01:15.512656 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1015,"thread_start_us":99,"threads_started":1}
I20260924 03:01:15.512646 18674 tablet_service.cc:1467] Processing CreateTablet for tablet e71200e6971e473087d26e7280851b60 (DEFAULT_TABLE table=test-table [id=de2dd2dae5d44c88af649bcaf1de5cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:15.512959 18674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71200e6971e473087d26e7280851b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:15.513294 18808 tablet_service.cc:1467] Processing CreateTablet for tablet e71200e6971e473087d26e7280851b60 (DEFAULT_TABLE table=test-table [id=de2dd2dae5d44c88af649bcaf1de5cb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:15.513581 18808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e71200e6971e473087d26e7280851b60. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:15.514868 18888 tablet_bootstrap.cc:492] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9: Bootstrap starting.
I20260924 03:01:15.515470 18888 tablet_bootstrap.cc:654] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:15.515930 18888 log.cc:824] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:15.516069 18889 tablet_bootstrap.cc:492] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d: Bootstrap starting.
I20260924 03:01:15.516506 18891 tablet_bootstrap.cc:492] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07: Bootstrap starting.
I20260924 03:01:15.516703 18888 tablet_bootstrap.cc:492] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9: No bootstrap required, opened a new log
I20260924 03:01:15.516707 18889 tablet_bootstrap.cc:654] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:15.516770 18888 ts_tablet_manager.cc:1397] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:01:15.517189 18891 tablet_bootstrap.cc:654] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:15.517221 18889 log.cc:824] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:15.517657 18891 log.cc:824] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:15.517961 18889 tablet_bootstrap.cc:492] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d: No bootstrap required, opened a new log
I20260924 03:01:15.518023 18889 ts_tablet_manager.cc:1397] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:01:15.518373 18891 tablet_bootstrap.cc:492] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07: No bootstrap required, opened a new log
I20260924 03:01:15.518314 18888 raft_consensus.cc:348] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2ce31cc30f4311bb78221838288a07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35747 } } peers { permanent_uuid: "dd1783752056447ca63dbdcb0181fbf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "6c1ff8afd8094f7fbb4a7abf84ec678d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260924 03:01:15.518438 18891 ts_tablet_manager.cc:1397] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:01:15.518451 18888 raft_consensus.cc:374] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:15.518491 18888 raft_consensus.cc:729] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd1783752056447ca63dbdcb0181fbf9, State: Initialized, Role: FOLLOWER
I20260924 03:01:15.518623 18888 consensus_queue.cc:260] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2ce31cc30f4311bb78221838288a07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35747 } } peers { permanent_uuid: "dd1783752056447ca63dbdcb0181fbf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "6c1ff8afd8094f7fbb4a7abf84ec678d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260924 03:01:15.518841 18888 ts_tablet_manager.cc:1428] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:01:15.518852 18606 heartbeater.cc:503] Master 127.0.98.62:44095 was elected leader, sending a full tablet report...
I20260924 03:01:15.520018 18889 raft_consensus.cc:348] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2ce31cc30f4311bb78221838288a07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35747 } } peers { permanent_uuid: "dd1783752056447ca63dbdcb0181fbf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "6c1ff8afd8094f7fbb4a7abf84ec678d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260924 03:01:15.520226 18889 raft_consensus.cc:374] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:15.520278 18889 raft_consensus.cc:729] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c1ff8afd8094f7fbb4a7abf84ec678d, State: Initialized, Role: FOLLOWER
I20260924 03:01:15.520258 18891 raft_consensus.cc:348] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2ce31cc30f4311bb78221838288a07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35747 } } peers { permanent_uuid: "dd1783752056447ca63dbdcb0181fbf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "6c1ff8afd8094f7fbb4a7abf84ec678d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260924 03:01:15.520408 18891 raft_consensus.cc:374] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:15.520403 18889 consensus_queue.cc:260] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2ce31cc30f4311bb78221838288a07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35747 } } peers { permanent_uuid: "dd1783752056447ca63dbdcb0181fbf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "6c1ff8afd8094f7fbb4a7abf84ec678d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260924 03:01:15.520445 18891 raft_consensus.cc:729] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1d2ce31cc30f4311bb78221838288a07, State: Initialized, Role: FOLLOWER
I20260924 03:01:15.520542 18891 consensus_queue.cc:260] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2ce31cc30f4311bb78221838288a07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35747 } } peers { permanent_uuid: "dd1783752056447ca63dbdcb0181fbf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "6c1ff8afd8094f7fbb4a7abf84ec678d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260924 03:01:15.520663 18740 heartbeater.cc:503] Master 127.0.98.62:44095 was elected leader, sending a full tablet report...
I20260924 03:01:15.520661 18889 ts_tablet_manager.cc:1428] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:01:15.520896 18874 heartbeater.cc:503] Master 127.0.98.62:44095 was elected leader, sending a full tablet report...
I20260924 03:01:15.520905 18891 ts_tablet_manager.cc:1428] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 03:01:15.566411 18894 raft_consensus.cc:482] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:01:15.566596 18894 raft_consensus.cc:504] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2ce31cc30f4311bb78221838288a07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35747 } } peers { permanent_uuid: "dd1783752056447ca63dbdcb0181fbf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "6c1ff8afd8094f7fbb4a7abf84ec678d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260924 03:01:15.566957 18894 leader_election.cc:288] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1d2ce31cc30f4311bb78221838288a07 (127.0.98.3:35747), 6c1ff8afd8094f7fbb4a7abf84ec678d (127.0.98.2:46263)
W20260924 03:01:15.568204 18741 tablet.cc:2369] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:01:15.569940 18885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.567244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56455 (local address 127.0.98.2:46263)
0924 03:01:15.567288 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:15.567289 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:15.567422 (+ 133us) server_negotiation.cc:407] Connection header received
0924 03:01:15.567447 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:15.567447 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:15.567457 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:15.567502 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:15.567616 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.568310 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.569260 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.569628 (+ 368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.569685 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.569695 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:15.569834 (+ 139us) server_negotiation.cc:299] Negotiation successful
0924 03:01:15.569872 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:01:15.569947 18884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.567188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:55325 (local address 127.0.98.3:35747)
0924 03:01:15.567244 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:15.567246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:15.567272 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:01:15.567299 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:15.567300 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:15.567309 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:15.567361 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:15.567515 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.568311 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.569244 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.569620 (+ 376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.569679 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.569690 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:15.569832 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 03:01:15.569883 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 03:01:15.569954 18608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.567123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:35747 (local address 127.0.98.1:55325)
0924 03:01:15.567171 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:15.567183 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:15.567228 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:15.567378 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:15.567379 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:15.567380 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:15.567451 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.567453 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.568406 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.568407 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.569154 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.569156 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.569692 (+ 536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.569693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.569771 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.569774 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:15.569817 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 03:01:15.569857 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260924 03:01:15.569967 18897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.567126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:46263 (local address 127.0.98.1:56455)
0924 03:01:15.567346 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:15.567355 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:15.567403 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:15.567513 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:15.567514 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:15.567515 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:15.567564 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.567565 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.568408 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.568409 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.569175 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.569177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.569703 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.569704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.569771 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.569774 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:15.569816 (+ 42us) client_negotiation.cc:240] Negotiation successful
0924 03:01:15.569859 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":207,"mutex_wait_us":64,"thread_start_us":110,"threads_started":1}
I20260924 03:01:15.570345 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71200e6971e473087d26e7280851b60" candidate_uuid: "dd1783752056447ca63dbdcb0181fbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d2ce31cc30f4311bb78221838288a07" is_pre_election: true
I20260924 03:01:15.570340 18694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71200e6971e473087d26e7280851b60" candidate_uuid: "dd1783752056447ca63dbdcb0181fbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c1ff8afd8094f7fbb4a7abf84ec678d" is_pre_election: true
I20260924 03:01:15.570469 18828 raft_consensus.cc:2454] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd1783752056447ca63dbdcb0181fbf9 in term 0.
I20260924 03:01:15.570477 18694 raft_consensus.cc:2454] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dd1783752056447ca63dbdcb0181fbf9 in term 0.
I20260924 03:01:15.570708 18493 leader_election.cc:302] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c1ff8afd8094f7fbb4a7abf84ec678d, dd1783752056447ca63dbdcb0181fbf9; no voters:
I20260924 03:01:15.570839 18894 raft_consensus.cc:2781] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:01:15.570904 18894 raft_consensus.cc:482] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:01:15.570922 18894 raft_consensus.cc:3037] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:15.571555 18894 raft_consensus.cc:504] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2ce31cc30f4311bb78221838288a07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35747 } } peers { permanent_uuid: "dd1783752056447ca63dbdcb0181fbf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "6c1ff8afd8094f7fbb4a7abf84ec678d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260924 03:01:15.571666 18894 leader_election.cc:288] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [CANDIDATE]: Term 1 election: Requested vote from peers 1d2ce31cc30f4311bb78221838288a07 (127.0.98.3:35747), 6c1ff8afd8094f7fbb4a7abf84ec678d (127.0.98.2:46263)
I20260924 03:01:15.571811 18694 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71200e6971e473087d26e7280851b60" candidate_uuid: "dd1783752056447ca63dbdcb0181fbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c1ff8afd8094f7fbb4a7abf84ec678d"
I20260924 03:01:15.571831 18828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e71200e6971e473087d26e7280851b60" candidate_uuid: "dd1783752056447ca63dbdcb0181fbf9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1d2ce31cc30f4311bb78221838288a07"
I20260924 03:01:15.571894 18694 raft_consensus.cc:3037] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:15.571894 18828 raft_consensus.cc:3037] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:15.572702 18694 raft_consensus.cc:2454] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd1783752056447ca63dbdcb0181fbf9 in term 1.
I20260924 03:01:15.572702 18828 raft_consensus.cc:2454] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dd1783752056447ca63dbdcb0181fbf9 in term 1.
I20260924 03:01:15.572853 18493 leader_election.cc:302] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c1ff8afd8094f7fbb4a7abf84ec678d, dd1783752056447ca63dbdcb0181fbf9; no voters:
I20260924 03:01:15.572973 18894 raft_consensus.cc:2781] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:15.573128 18894 raft_consensus.cc:686] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [term 1 LEADER]: Becoming Leader. State: Replica: dd1783752056447ca63dbdcb0181fbf9, State: Running, Role: LEADER
I20260924 03:01:15.573216 18894 consensus_queue.cc:237] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2ce31cc30f4311bb78221838288a07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35747 } } peers { permanent_uuid: "dd1783752056447ca63dbdcb0181fbf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } } peers { permanent_uuid: "6c1ff8afd8094f7fbb4a7abf84ec678d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } }
I20260924 03:01:15.574004 18415 catalog_manager.cc:5495] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 reported cstate change: term changed from 0 to 1, leader changed from <none> to dd1783752056447ca63dbdcb0181fbf9 (127.0.98.1). New cstate: current_term: 1 leader_uuid: "dd1783752056447ca63dbdcb0181fbf9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1d2ce31cc30f4311bb78221838288a07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd1783752056447ca63dbdcb0181fbf9" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 41071 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c1ff8afd8094f7fbb4a7abf84ec678d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 } health_report { overall_health: UNKNOWN } } }
I20260924 03:01:15.585315 18887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.583357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38982 (local address 127.0.98.1:41071)
0924 03:01:15.583412 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:15.583414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:15.583421 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:01:15.583469 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:15.583470 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:15.583486 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:15.583540 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:01:15.583626 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.584082 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.584746 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.584931 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.584980 (+ 49us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.585073 (+ 93us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:01:15.585154 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:01:15.585216 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:15.585237 (+ 21us) server_negotiation.cc:299] Negotiation successful
0924 03:01:15.585245 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 03:01:15.586910 18694 raft_consensus.cc:1261] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d [term 1 FOLLOWER]: Refusing update from remote peer dd1783752056447ca63dbdcb0181fbf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:01:15.586912 18828 raft_consensus.cc:1261] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07 [term 1 FOLLOWER]: Refusing update from remote peer dd1783752056447ca63dbdcb0181fbf9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:01:15.587234 18898 consensus_queue.cc:1035] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c1ff8afd8094f7fbb4a7abf84ec678d" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 46263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:01:15.587348 18894 consensus_queue.cc:1035] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1d2ce31cc30f4311bb78221838288a07" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 35747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:01:15.590308 18902 mvcc.cc:205] Tried to move back new op lower bound from 7332736514401366016 to 7332736514348404736. Current Snapshot: MvccSnapshot[applied={T|T < 7332736514401366016}]
I20260924 03:01:15.591918 18415 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:01:15.592018 18415 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36980:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:01:15.592700 18415 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d44ef2dac6be44e3a0fa9bb5ad7b382c: Sending DeleteTablet for 3 replicas of tablet e71200e6971e473087d26e7280851b60
I20260924 03:01:15.593003 18808 tablet_service.cc:1514] Processing DeleteTablet for tablet e71200e6971e473087d26e7280851b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:56674
I20260924 03:01:15.593020 18674 tablet_service.cc:1514] Processing DeleteTablet for tablet e71200e6971e473087d26e7280851b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41560
I20260924 03:01:15.593209 18540 tablet_service.cc:1514] Processing DeleteTablet for tablet e71200e6971e473087d26e7280851b60 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:15 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:38970
I20260924 03:01:15.593408 18912 tablet_replica.cc:330] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9: stopping tablet replica
I20260924 03:01:15.593513 18912 raft_consensus.cc:2229] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:01:15.593588 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 18477
I20260924 03:01:15.593603 18912 raft_consensus.cc:2258] T e71200e6971e473087d26e7280851b60 P dd1783752056447ca63dbdcb0181fbf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:01:15.595041 18910 tablet_replica.cc:330] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d: stopping tablet replica
I20260924 03:01:15.595117 18910 raft_consensus.cc:2229] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:01:15.595183 18910 raft_consensus.cc:2258] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:01:15.595302 18911 tablet_replica.cc:330] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07: stopping tablet replica
I20260924 03:01:15.595391 18911 raft_consensus.cc:2229] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:01:15.595487 18911 raft_consensus.cc:2258] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:01:15.595623 18910 ts_tablet_manager.cc:1905] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:01:15.595885 18911 ts_tablet_manager.cc:1905] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:01:15.597371 18910 ts_tablet_manager.cc:1918] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:01:15.597371 18911 ts_tablet_manager.cc:1918] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:01:15.597443 18911 log.cc:1196] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/e71200e6971e473087d26e7280851b60
I20260924 03:01:15.597443 18910 log.cc:1196] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/e71200e6971e473087d26e7280851b60
I20260924 03:01:15.597707 18911 ts_tablet_manager.cc:1939] T e71200e6971e473087d26e7280851b60 P 1d2ce31cc30f4311bb78221838288a07: Deleting consensus metadata
I20260924 03:01:15.597707 18910 ts_tablet_manager.cc:1939] T e71200e6971e473087d26e7280851b60 P 6c1ff8afd8094f7fbb4a7abf84ec678d: Deleting consensus metadata
I20260924 03:01:15.598075 18401 catalog_manager.cc:4841] TS 1d2ce31cc30f4311bb78221838288a07 (127.0.98.3:35747): tablet e71200e6971e473087d26e7280851b60 (table test-table [id=de2dd2dae5d44c88af649bcaf1de5cb8]) successfully deleted
I20260924 03:01:15.598096 18399 catalog_manager.cc:4841] TS 6c1ff8afd8094f7fbb4a7abf84ec678d (127.0.98.2:46263): tablet e71200e6971e473087d26e7280851b60 (table test-table [id=de2dd2dae5d44c88af649bcaf1de5cb8]) successfully deleted
W20260924 03:01:15.598668 18401 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:41071 (error 108)
I20260924 03:01:15.598830 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 18611
I20260924 03:01:15.599053 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.598953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:41071 (local address 127.0.0.1:38990)
0924 03:01:15.598998 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:15.599026 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41071: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 03:01:15.599139 18401 catalog_manager.cc:4584] TS dd1783752056447ca63dbdcb0181fbf9 (127.0.98.1:41071): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e71200e6971e473087d26e7280851b60: Network error: Client connection negotiation failed: client connection to 127.0.98.1:41071: connect: Connection refused (error 111)
I20260924 03:01:15.603462 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 18745
I20260924 03:01:15.607582 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 18383
2026-09-24T03:01:15Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1811 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-24T03:01:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:15Z Disabled control of system clock
I20260924 03:01:15.632195 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45309
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:41039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45309
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:01:15.711638 18917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:15.711851 18917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:15.711871 18917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:15.713016 18917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:15.713058 18917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:15.713070 18917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:15.713083 18917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:15.713093 18917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:15.714421 18917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45309
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:15.714679 18917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:15.714964 18917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:15.719650 18925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:15.719645 18923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:15.719646 18922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:15.719673 18917 server_base.cc:1034] running on GCE node
I20260924 03:01:15.720073 18917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:15.720332 18917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:15.721480 18917 hybrid_clock.cc:648] HybridClock initialized: now 1790218875721454 us; error 29 us; skew 500 ppm
I20260924 03:01:15.722903 18917 webserver.cc:460] Webserver started at http://127.0.98.62:43685/ using document root <none> and password file <none>
I20260924 03:01:15.723105 18917 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:15.723147 18917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:15.723240 18917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:15.724265 18917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "9b4aed1cc6ae40ed940600789126c5f6"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "fcda070c5bcdb8623d91ebb437cd306f"
server_key_iv: "a3240bd7efbb1889b83724dedfc529c6"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.724555 18917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "9b4aed1cc6ae40ed940600789126c5f6"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "fcda070c5bcdb8623d91ebb437cd306f"
server_key_iv: "a3240bd7efbb1889b83724dedfc529c6"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.725742 18917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:01:15.726434 18931 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:15.726606 18917 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:01:15.726688 18917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "9b4aed1cc6ae40ed940600789126c5f6"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "fcda070c5bcdb8623d91ebb437cd306f"
server_key_iv: "a3240bd7efbb1889b83724dedfc529c6"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.726750 18917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:15.737489 18917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:15.737789 18917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:15.737920 18917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:15.741766 18917 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45309
I20260924 03:01:15.741798 18983 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45309 every 8 connection(s)
I20260924 03:01:15.742106 18917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:15.743039 18984 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:15.745276 18984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6: Bootstrap starting.
I20260924 03:01:15.745903 18984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:15.745970 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 18917
I20260924 03:01:15.746097 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:15.746205 18984 log.cc:824] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:15.746300 392 external_mini_cluster.cc:1428] Setting key d6f02d2671e7924817bbc19e1de71a45
I20260924 03:01:15.746913 18984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6: No bootstrap required, opened a new log
I20260924 03:01:15.748638 18984 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4aed1cc6ae40ed940600789126c5f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260924 03:01:15.748765 18984 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:15.748796 18984 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b4aed1cc6ae40ed940600789126c5f6, State: Initialized, Role: FOLLOWER
I20260924 03:01:15.748920 18984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4aed1cc6ae40ed940600789126c5f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260924 03:01:15.749001 18984 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:15.749032 18984 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:15.749078 18984 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:15.749598 18984 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4aed1cc6ae40ed940600789126c5f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260924 03:01:15.749686 18984 leader_election.cc:302] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9b4aed1cc6ae40ed940600789126c5f6; no voters:
I20260924 03:01:15.749832 18984 leader_election.cc:288] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:15.749881 18989 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:15.750052 18989 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [term 1 LEADER]: Becoming Leader. State: Replica: 9b4aed1cc6ae40ed940600789126c5f6, State: Running, Role: LEADER
I20260924 03:01:15.750175 18984 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:15.750178 18989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4aed1cc6ae40ed940600789126c5f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } }
I20260924 03:01:15.750615 18990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9b4aed1cc6ae40ed940600789126c5f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4aed1cc6ae40ed940600789126c5f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } } }
I20260924 03:01:15.750706 18990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:15.750624 18991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9b4aed1cc6ae40ed940600789126c5f6. Latest consensus state: current_term: 1 leader_uuid: "9b4aed1cc6ae40ed940600789126c5f6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9b4aed1cc6ae40ed940600789126c5f6" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45309 } } }
I20260924 03:01:15.750805 18991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6 [sys.catalog]: This master's current role is: LEADER
W20260924 03:01:15.751907 19005 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:01:15.751960 19005 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:01:15.752012 18996 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:15.752200 18996 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:15.753082 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.746785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51430 (local address 127.0.98.62:45309)
0924 03:01:15.746988 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:15.747004 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:15.747036 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:01:15.747994 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:15.748021 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:15.748081 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:15.748204 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:15.748415 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.749781 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.750893 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.751480 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.751598 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.752384 (+ 786us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:15.752388 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:15.752388 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:15.752592 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:15.752636 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:15.752651 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:15.752716 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:15.752789 (+ 73us) server_negotiation.cc:299] Negotiation successful
0924 03:01:15.752868 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260924 03:01:15.753686 18996 catalog_manager.cc:1347] Generated new cluster ID: 37d913948b9449f6a1ef289f7ad5e651
I20260924 03:01:15.753727 18996 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:15.765413 18996 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:15.765918 18996 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:15.771431 18996 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6: Generated new TSK 0
I20260924 03:01:15.771631 18996 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:15.773036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--builtin_ntp_servers=127.0.98.20:41039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:01:15.856292 19008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:15.856503 19008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:15.856531 19008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:15.857703 19008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:15.857743 19008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:15.857776 19008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:01:15.859261 19008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19008
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:15.859498 19008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:15.859784 19008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:15.864549 19016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:15.865455 19014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:15.865531 19008 server_base.cc:1034] running on GCE node
W20260924 03:01:15.865499 19013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:15.865799 19008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:15.866065 19008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:15.867233 19008 hybrid_clock.cc:648] HybridClock initialized: now 1790218875867213 us; error 35 us; skew 500 ppm
I20260924 03:01:15.868579 19008 webserver.cc:460] Webserver started at http://127.0.98.1:41095/ using document root <none> and password file <none>
I20260924 03:01:15.868799 19008 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:15.868847 19008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:15.868979 19008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:15.870021 19008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "0d3acdc669f44938b3e20f4284d13d61"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "0dfbbb4eff69b87c52bf896493463583"
server_key_iv: "cd7d48a0f162586dbeab47274f8abd92"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.870355 19008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "0d3acdc669f44938b3e20f4284d13d61"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "0dfbbb4eff69b87c52bf896493463583"
server_key_iv: "cd7d48a0f162586dbeab47274f8abd92"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.871591 19008 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:01:15.872273 19022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:15.872418 19008 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:15.872498 19008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "0d3acdc669f44938b3e20f4284d13d61"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "0dfbbb4eff69b87c52bf896493463583"
server_key_iv: "cd7d48a0f162586dbeab47274f8abd92"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.872565 19008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:15.883164 19008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:15.883441 19008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:15.883561 19008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:15.883762 19008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:15.884080 19008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:15.884114 19008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:15.884141 19008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:15.884166 19008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:15.889460 19008 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:36865
I20260924 03:01:15.889550 19135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:36865 every 8 connection(s)
I20260924 03:01:15.889812 19008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:01:15.894950 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.890558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:56277 (local address 127.0.98.62:45309)
0924 03:01:15.890599 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:15.890601 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:15.890810 (+ 209us) server_negotiation.cc:407] Connection header received
0924 03:01:15.891333 (+ 523us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:15.891335 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:15.891348 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:15.891400 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:15.892073 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.892583 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.893603 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.893803 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.893859 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.894699 (+ 840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:15.894700 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:15.894701 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:15.894705 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:15.894722 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:15.894725 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:15.894777 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:15.894845 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 03:01:15.894872 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:01:15.895148 19138 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:15.890500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.1:56277)
0924 03:01:15.890695 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:15.890729 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:15.891230 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:15.891473 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:15.891477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:15.891488 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:15.891983 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.891987 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.892670 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.892671 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.893512 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:15.893515 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:15.893876 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:15.893876 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:15.894013 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:15.894566 (+ 553us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:01:15.894567 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:15.894596 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:01:15.894617 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:15.894778 (+ 161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:15.894783 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:15.894786 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:15.894845 (+ 59us) client_negotiation.cc:240] Negotiation successful
0924 03:01:15.894944 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":75,"threads_started":1}
I20260924 03:01:15.895599 19136 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45309
I20260924 03:01:15.895687 19136 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:15.895853 19136 heartbeater.cc:511] Master 127.0.98.62:45309 requested a full tablet report, sending...
I20260924 03:01:15.896317 18948 ts_manager.cc:194] Registered new tserver with Master: 0d3acdc669f44938b3e20f4284d13d61 (127.0.98.1:36865)
I20260924 03:01:15.897025 18948 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:56277
I20260924 03:01:15.897955 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 19008
I20260924 03:01:15.898051 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:01:15.898186 392 external_mini_cluster.cc:1428] Setting key 27d19164d54392567895a34eb96c1fa9
I20260924 03:01:15.899034 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--builtin_ntp_servers=127.0.98.20:41039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:01:15.977571 19139 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:15.977763 19139 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:15.977783 19139 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:15.978875 19139 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:15.978906 19139 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:15.978929 19139 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:01:15.980195 19139 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19139
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:15.980376 19139 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:15.980619 19139 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:15.985402 19147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:15.985502 19144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:15.985502 19145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:15.985595 19139 server_base.cc:1034] running on GCE node
I20260924 03:01:15.985860 19139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:15.986081 19139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:15.987217 19139 hybrid_clock.cc:648] HybridClock initialized: now 1790218875987194 us; error 39 us; skew 500 ppm
I20260924 03:01:15.988351 19139 webserver.cc:460] Webserver started at http://127.0.98.2:40653/ using document root <none> and password file <none>
I20260924 03:01:15.988638 19139 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:15.988701 19139 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:15.988795 19139 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:15.989776 19139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "c798c769d59749778f6ef9df5321ce8c"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "5693df6bb1b009b940bef1ce3311efc7"
server_key_iv: "ff76b378fd0263f66a9f08dbe390470c"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.990056 19139 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "c798c769d59749778f6ef9df5321ce8c"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "5693df6bb1b009b940bef1ce3311efc7"
server_key_iv: "ff76b378fd0263f66a9f08dbe390470c"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.991282 19139 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:01:15.991941 19153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:15.992093 19139 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:01:15.992170 19139 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "c798c769d59749778f6ef9df5321ce8c"
format_stamp: "Formatted at 2026-09-24 03:01:15 on dist-test-slave-sb6p"
server_key: "5693df6bb1b009b940bef1ce3311efc7"
server_key_iv: "ff76b378fd0263f66a9f08dbe390470c"
server_key_version: "encryptionkey@0"
I20260924 03:01:15.992223 19139 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:16.014052 19139 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:16.014325 19139 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:16.014423 19139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:16.014647 19139 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:16.014971 19139 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:16.015010 19139 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:16.015040 19139 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:16.015057 19139 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:16.020432 19139 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:43355
I20260924 03:01:16.020479 19266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:43355 every 8 connection(s)
I20260924 03:01:16.020812 19139 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:01:16.023391 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 19139
I20260924 03:01:16.023479 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:01:16.023643 392 external_mini_cluster.cc:1428] Setting key 7cb9f5419b9a23936a94dbe4193bc5ed
I20260924 03:01:16.024554 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--builtin_ntp_servers=127.0.98.20:41039
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:01:16.026156 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:16.021512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:37073 (local address 127.0.98.62:45309)
0924 03:01:16.021560 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:16.021562 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:16.021803 (+ 241us) server_negotiation.cc:407] Connection header received
0924 03:01:16.022272 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:16.022273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:16.022283 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:16.022343 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:16.023012 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.023636 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.024778 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.024963 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.025016 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:16.025914 (+ 898us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:16.025915 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:16.025915 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:16.025920 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:16.025938 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:16.025942 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:16.026001 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:16.026053 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 03:01:16.026077 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:01:16.026310 19269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:16.021470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.2:37073)
0924 03:01:16.021653 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:16.021707 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:16.022184 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:16.022397 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:16.022400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:16.022408 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:16.022932 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:16.022938 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.023729 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.023731 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:16.024712 (+ 981us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:16.024715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.025038 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.025039 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:16.025189 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:16.025750 (+ 561us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:01:16.025751 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:16.025795 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:01:16.025826 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:16.026002 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:16.026005 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:16.026007 (+ 2us) client_negotiation.cc:769] Sending connection context
0924 03:01:16.026052 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 03:01:16.026130 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":77,"threads_started":1}
I20260924 03:01:16.026810 19267 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45309
I20260924 03:01:16.026899 19267 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:16.027066 19267 heartbeater.cc:511] Master 127.0.98.62:45309 requested a full tablet report, sending...
I20260924 03:01:16.027402 18948 ts_manager.cc:194] Registered new tserver with Master: c798c769d59749778f6ef9df5321ce8c (127.0.98.2:43355)
I20260924 03:01:16.027753 18948 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:37073
W20260924 03:01:16.104066 19270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:16.104255 19270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:16.104274 19270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:16.105409 19270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:16.105453 19270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:16.105476 19270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:01:16.106848 19270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:41039
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:45309
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19270
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:16.107079 19270 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:16.107354 19270 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:16.112124 19275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:16.112160 19276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:16.112177 19278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:16.112299 19270 server_base.cc:1034] running on GCE node
I20260924 03:01:16.112535 19270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:16.112726 19270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:16.113855 19270 hybrid_clock.cc:648] HybridClock initialized: now 1790218876113843 us; error 27 us; skew 500 ppm
I20260924 03:01:16.115017 19270 webserver.cc:460] Webserver started at http://127.0.98.3:42165/ using document root <none> and password file <none>
I20260924 03:01:16.115231 19270 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:16.115279 19270 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:16.115402 19270 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:16.116420 19270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "83c5d539bbe34a1a9b421ee66a621b33"
format_stamp: "Formatted at 2026-09-24 03:01:16 on dist-test-slave-sb6p"
server_key: "e7fe1a99e7ee955cf6ca09b6b13aae48"
server_key_iv: "9a6576b2a7b1a18f13e4f89f73d91e47"
server_key_version: "encryptionkey@0"
I20260924 03:01:16.116734 19270 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "83c5d539bbe34a1a9b421ee66a621b33"
format_stamp: "Formatted at 2026-09-24 03:01:16 on dist-test-slave-sb6p"
server_key: "e7fe1a99e7ee955cf6ca09b6b13aae48"
server_key_iv: "9a6576b2a7b1a18f13e4f89f73d91e47"
server_key_version: "encryptionkey@0"
I20260924 03:01:16.117843 19270 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:01:16.118451 19284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:16.118600 19270 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:16.118677 19270 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "83c5d539bbe34a1a9b421ee66a621b33"
format_stamp: "Formatted at 2026-09-24 03:01:16 on dist-test-slave-sb6p"
server_key: "e7fe1a99e7ee955cf6ca09b6b13aae48"
server_key_iv: "9a6576b2a7b1a18f13e4f89f73d91e47"
server_key_version: "encryptionkey@0"
I20260924 03:01:16.118737 19270 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:16.142366 19270 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:16.142699 19270 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:16.142829 19270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:16.143038 19270 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:16.143386 19270 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:16.143435 19270 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:16.143465 19270 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:16.143491 19270 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:16.148821 19270 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:34003
I20260924 03:01:16.148885 19397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:34003 every 8 connection(s)
I20260924 03:01:16.149174 19270 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:01:16.149601 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 19270
I20260924 03:01:16.149684 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:01:16.149861 392 external_mini_cluster.cc:1428] Setting key cdd430b3cdc4bf76dce0239c9b108462
I20260924 03:01:16.154747 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:16.150389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:42537 (local address 127.0.98.62:45309)
0924 03:01:16.150437 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:16.150438 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:16.150746 (+ 308us) server_negotiation.cc:407] Connection header received
0924 03:01:16.151281 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:16.151282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:16.151293 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:16.151345 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:16.152022 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.152511 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.153472 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.153669 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.153720 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:16.154477 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:16.154479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:16.154479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:16.154484 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:16.154500 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:16.154503 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:16.154568 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:16.154639 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 03:01:16.154666 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:01:16.154937 19400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:16.150431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45309 (local address 127.0.98.3:42537)
0924 03:01:16.150639 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:16.150674 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:16.151196 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:16.151451 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:16.151455 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:16.151463 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:16.151940 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:16.151945 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.152594 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.152595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:16.153380 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:16.153383 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.153732 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.153733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:16.153859 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:16.154348 (+ 489us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:01:16.154349 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:16.154377 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:01:16.154400 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:16.154571 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:16.154575 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:16.154578 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:16.154638 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 03:01:16.154742 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":106,"threads_started":1}
I20260924 03:01:16.155354 19398 heartbeater.cc:347] Connected to a master server at 127.0.98.62:45309
I20260924 03:01:16.155441 19398 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:16.155582 19398 heartbeater.cc:511] Master 127.0.98.62:45309 requested a full tablet report, sending...
I20260924 03:01:16.155896 18948 ts_manager.cc:194] Registered new tserver with Master: 83c5d539bbe34a1a9b421ee66a621b33 (127.0.98.3:34003)
I20260924 03:01:16.156219 18948 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:42537
I20260924 03:01:16.161573 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:01:16.165745 18988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:16.162888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51440 (local address 127.0.98.62:45309)
0924 03:01:16.162971 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:16.162974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:16.163075 (+ 101us) server_negotiation.cc:407] Connection header received
0924 03:01:16.163144 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:16.163144 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:16.163154 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:16.163194 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:16.163403 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.163886 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.164603 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.164786 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.164843 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:16.165484 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:16.165486 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:16.165486 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:16.165490 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:16.165506 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:16.165509 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:16.165581 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:16.165650 (+ 69us) server_negotiation.cc:299] Negotiation successful
0924 03:01:16.165660 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260924 03:01:16.167240 18948 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:01:16.168807 18948 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:01:16.174929 19406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:16.170741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:43355 (local address 127.0.0.1:51890)
0924 03:01:16.171036 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:16.171075 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:16.171127 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:16.171704 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:16.171705 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:16.171707 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:16.171742 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:16.171743 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.172982 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.172983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:16.173890 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:16.173892 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.174688 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.174689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:16.174749 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:16.174761 (+ 12us) client_negotiation.cc:769] Sending connection context
0924 03:01:16.174834 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 03:01:16.174875 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":277,"thread_start_us":87,"threads_started":1}
I20260924 03:01:16.174970 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:16.170848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51890 (local address 127.0.98.2:43355)
0924 03:01:16.171483 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:16.171487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:16.171496 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:01:16.171559 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:16.171561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:16.171576 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:16.171643 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:16.171850 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.172912 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.173959 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.174617 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.174685 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:16.174773 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:16.174848 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 03:01:16.174885 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":596,"thread_start_us":79,"threads_started":1}
I20260924 03:01:16.175556 19407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:16.170728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36865 (local address 127.0.0.1:36706)
0924 03:01:16.171036 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:16.171076 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:16.171131 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:16.171369 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:16.171372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:16.171374 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:16.171437 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:16.171439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.172805 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.172806 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:16.174138 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:16.174140 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.174991 (+ 851us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.174992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:16.175261 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:16.175265 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:01:16.175447 (+ 182us) client_negotiation.cc:240] Negotiation successful
0924 03:01:16.175490 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":84,"thread_start_us":56,"threads_started":1}
I20260924 03:01:16.175586 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:16.170823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36706 (local address 127.0.98.1:36865)
0924 03:01:16.171235 (+ 412us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:16.171239 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:16.171248 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:01:16.171282 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:16.171283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:16.171295 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:16.171359 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:16.171600 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.172724 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.174217 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.174916 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.175109 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:16.175195 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:16.175467 (+ 272us) server_negotiation.cc:299] Negotiation successful
0924 03:01:16.175508 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":80,"threads_started":1}
I20260924 03:01:16.175698 19201 tablet_service.cc:1467] Processing CreateTablet for tablet 555d9770c2d04b5a8cd47fb30a081d80 (DEFAULT_TABLE table=test-table [id=3c8f3ce95ffa40929e97819adfedc7a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:16.176059 19201 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 555d9770c2d04b5a8cd47fb30a081d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:16.176415 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:16.171022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39476 (local address 127.0.98.3:34003)
0924 03:01:16.171200 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:16.171204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:16.171834 (+ 630us) server_negotiation.cc:407] Connection header received
0924 03:01:16.171873 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:16.171875 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:16.171886 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:16.171944 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:16.172045 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.172744 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.175228 (+ 2484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.176105 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.176163 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:16.176240 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:16.176300 (+ 60us) server_negotiation.cc:299] Negotiation successful
0924 03:01:16.176335 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":88,"threads_started":1}
I20260924 03:01:16.176317 19070 tablet_service.cc:1467] Processing CreateTablet for tablet 555d9770c2d04b5a8cd47fb30a081d80 (DEFAULT_TABLE table=test-table [id=3c8f3ce95ffa40929e97819adfedc7a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:16.176442 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:16.170915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34003 (local address 127.0.0.1:39476)
0924 03:01:16.171730 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:16.171740 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:16.171784 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:16.171950 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:16.171951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:16.171953 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:16.171988 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:16.171989 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.172823 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.172824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:16.174133 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:16.174136 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.176178 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.176179 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:16.176239 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:16.176243 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:01:16.176290 (+ 47us) client_negotiation.cc:240] Negotiation successful
0924 03:01:16.176338 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":799,"mutex_wait_us":35,"thread_start_us":62,"threads_started":1}
I20260924 03:01:16.176608 19070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 555d9770c2d04b5a8cd47fb30a081d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:16.176954 19332 tablet_service.cc:1467] Processing CreateTablet for tablet 555d9770c2d04b5a8cd47fb30a081d80 (DEFAULT_TABLE table=test-table [id=3c8f3ce95ffa40929e97819adfedc7a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:16.177179 19332 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 555d9770c2d04b5a8cd47fb30a081d80. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:16.179432 19412 tablet_bootstrap.cc:492] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c: Bootstrap starting.
I20260924 03:01:16.179579 19414 tablet_bootstrap.cc:492] T 555d9770c2d04b5a8cd47fb30a081d80 P 0d3acdc669f44938b3e20f4284d13d61: Bootstrap starting.
I20260924 03:01:16.179987 19413 tablet_bootstrap.cc:492] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33: Bootstrap starting.
I20260924 03:01:16.180126 19412 tablet_bootstrap.cc:654] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:16.180171 19414 tablet_bootstrap.cc:654] T 555d9770c2d04b5a8cd47fb30a081d80 P 0d3acdc669f44938b3e20f4284d13d61: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:16.180393 19412 log.cc:824] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:16.180461 19414 log.cc:824] T 555d9770c2d04b5a8cd47fb30a081d80 P 0d3acdc669f44938b3e20f4284d13d61: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:16.180598 19413 tablet_bootstrap.cc:654] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:16.180898 19413 log.cc:824] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:16.181056 19412 tablet_bootstrap.cc:492] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c: No bootstrap required, opened a new log
I20260924 03:01:16.181119 19412 ts_tablet_manager.cc:1397] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:01:16.181128 19414 tablet_bootstrap.cc:492] T 555d9770c2d04b5a8cd47fb30a081d80 P 0d3acdc669f44938b3e20f4284d13d61: No bootstrap required, opened a new log
I20260924 03:01:16.181180 19414 ts_tablet_manager.cc:1397] T 555d9770c2d04b5a8cd47fb30a081d80 P 0d3acdc669f44938b3e20f4284d13d61: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:01:16.181511 19413 tablet_bootstrap.cc:492] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33: No bootstrap required, opened a new log
I20260924 03:01:16.181566 19413 ts_tablet_manager.cc:1397] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:01:16.183391 19413 raft_consensus.cc:348] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3acdc669f44938b3e20f4284d13d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36865 } } peers { permanent_uuid: "c798c769d59749778f6ef9df5321ce8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43355 } } peers { permanent_uuid: "83c5d539bbe34a1a9b421ee66a621b33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34003 } }
I20260924 03:01:16.183517 19413 raft_consensus.cc:374] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:16.183547 19413 raft_consensus.cc:729] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 83c5d539bbe34a1a9b421ee66a621b33, State: Initialized, Role: FOLLOWER
I20260924 03:01:16.183677 19413 consensus_queue.cc:260] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3acdc669f44938b3e20f4284d13d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36865 } } peers { permanent_uuid: "c798c769d59749778f6ef9df5321ce8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43355 } } peers { permanent_uuid: "83c5d539bbe34a1a9b421ee66a621b33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34003 } }
I20260924 03:01:16.183599 19414 raft_consensus.cc:348] T 555d9770c2d04b5a8cd47fb30a081d80 P 0d3acdc669f44938b3e20f4284d13d61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3acdc669f44938b3e20f4284d13d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36865 } } peers { permanent_uuid: "c798c769d59749778f6ef9df5321ce8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43355 } } peers { permanent_uuid: "83c5d539bbe34a1a9b421ee66a621b33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34003 } }
I20260924 03:01:16.183743 19414 raft_consensus.cc:374] T 555d9770c2d04b5a8cd47fb30a081d80 P 0d3acdc669f44938b3e20f4284d13d61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:16.183786 19414 raft_consensus.cc:729] T 555d9770c2d04b5a8cd47fb30a081d80 P 0d3acdc669f44938b3e20f4284d13d61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3acdc669f44938b3e20f4284d13d61, State: Initialized, Role: FOLLOWER
I20260924 03:01:16.183739 19412 raft_consensus.cc:348] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3acdc669f44938b3e20f4284d13d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36865 } } peers { permanent_uuid: "c798c769d59749778f6ef9df5321ce8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43355 } } peers { permanent_uuid: "83c5d539bbe34a1a9b421ee66a621b33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34003 } }
I20260924 03:01:16.183866 19412 raft_consensus.cc:374] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:16.183905 19412 raft_consensus.cc:729] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c798c769d59749778f6ef9df5321ce8c, State: Initialized, Role: FOLLOWER
I20260924 03:01:16.183904 19413 ts_tablet_manager.cc:1428] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260924 03:01:16.183921 19414 consensus_queue.cc:260] T 555d9770c2d04b5a8cd47fb30a081d80 P 0d3acdc669f44938b3e20f4284d13d61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3acdc669f44938b3e20f4284d13d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36865 } } peers { permanent_uuid: "c798c769d59749778f6ef9df5321ce8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43355 } } peers { permanent_uuid: "83c5d539bbe34a1a9b421ee66a621b33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34003 } }
I20260924 03:01:16.184017 19398 heartbeater.cc:503] Master 127.0.98.62:45309 was elected leader, sending a full tablet report...
I20260924 03:01:16.184052 19412 consensus_queue.cc:260] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3acdc669f44938b3e20f4284d13d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36865 } } peers { permanent_uuid: "c798c769d59749778f6ef9df5321ce8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43355 } } peers { permanent_uuid: "83c5d539bbe34a1a9b421ee66a621b33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34003 } }
I20260924 03:01:16.184159 19414 ts_tablet_manager.cc:1428] T 555d9770c2d04b5a8cd47fb30a081d80 P 0d3acdc669f44938b3e20f4284d13d61: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:01:16.184177 19136 heartbeater.cc:503] Master 127.0.98.62:45309 was elected leader, sending a full tablet report...
I20260924 03:01:16.184481 19412 ts_tablet_manager.cc:1428] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 03:01:16.184518 19267 heartbeater.cc:503] Master 127.0.98.62:45309 was elected leader, sending a full tablet report...
I20260924 03:01:16.220552 19420 raft_consensus.cc:482] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:01:16.220686 19420 raft_consensus.cc:504] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3acdc669f44938b3e20f4284d13d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36865 } } peers { permanent_uuid: "c798c769d59749778f6ef9df5321ce8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43355 } } peers { permanent_uuid: "83c5d539bbe34a1a9b421ee66a621b33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34003 } }
I20260924 03:01:16.221040 19420 leader_election.cc:288] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0d3acdc669f44938b3e20f4284d13d61 (127.0.98.1:36865), 83c5d539bbe34a1a9b421ee66a621b33 (127.0.98.3:34003)
I20260924 03:01:16.223891 19408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:16.221279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:59435 (local address 127.0.98.1:36865)
0924 03:01:16.221320 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:16.221322 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:16.221331 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:01:16.221393 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:16.221394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:16.221403 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:16.221446 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:16.221648 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.222204 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.223161 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.223566 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.223618 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:16.223627 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:16.223777 (+ 150us) server_negotiation.cc:299] Negotiation successful
0924 03:01:16.223817 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:01:16.223908 19411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:16.221392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:34229 (local address 127.0.98.3:34003)
0924 03:01:16.221471 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:16.221473 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:16.221482 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:01:16.221509 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:16.221510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:16.221518 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:16.221569 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:16.221721 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.222202 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.223201 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.223607 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.223676 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:16.223685 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:16.223809 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 03:01:16.223845 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260924 03:01:16.223908 19269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:16.221200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36865 (local address 127.0.98.2:59435)
0924 03:01:16.221235 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:16.221249 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:16.221325 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:16.221498 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:16.221499 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:16.221501 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:16.221570 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:16.221572 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.222281 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.222282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:16.223084 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:16.223086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.223646 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.223647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:16.223716 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:16.223720 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:01:16.223765 (+ 45us) client_negotiation.cc:240] Negotiation successful
0924 03:01:16.223822 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16}
I20260924 03:01:16.223927 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:16.221227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:34003 (local address 127.0.98.2:34229)
0924 03:01:16.221375 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:16.221385 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:16.221441 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:16.221580 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:16.221581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:16.221582 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:16.221658 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:16.221659 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.222275 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.222276 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:16.223119 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:16.223122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.223691 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.223692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:16.223744 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:16.223748 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:01:16.223798 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 03:01:16.223831 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":14,"thread_start_us":103,"threads_started":1}
I20260924 03:01:16.224283 19090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "555d9770c2d04b5a8cd47fb30a081d80" candidate_uuid: "c798c769d59749778f6ef9df5321ce8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3acdc669f44938b3e20f4284d13d61" is_pre_election: true
I20260924 03:01:16.224447 19090 raft_consensus.cc:2454] T 555d9770c2d04b5a8cd47fb30a081d80 P 0d3acdc669f44938b3e20f4284d13d61 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c798c769d59749778f6ef9df5321ce8c in term 0.
I20260924 03:01:16.224447 19352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "555d9770c2d04b5a8cd47fb30a081d80" candidate_uuid: "c798c769d59749778f6ef9df5321ce8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c5d539bbe34a1a9b421ee66a621b33" is_pre_election: true
I20260924 03:01:16.224596 19352 raft_consensus.cc:2454] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c798c769d59749778f6ef9df5321ce8c in term 0.
I20260924 03:01:16.224673 19156 leader_election.cc:302] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3acdc669f44938b3e20f4284d13d61, c798c769d59749778f6ef9df5321ce8c; no voters:
I20260924 03:01:16.224828 19420 raft_consensus.cc:2781] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:01:16.224885 19420 raft_consensus.cc:482] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:01:16.224918 19420 raft_consensus.cc:3037] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:16.225565 19420 raft_consensus.cc:504] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3acdc669f44938b3e20f4284d13d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36865 } } peers { permanent_uuid: "c798c769d59749778f6ef9df5321ce8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43355 } } peers { permanent_uuid: "83c5d539bbe34a1a9b421ee66a621b33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34003 } }
I20260924 03:01:16.225677 19420 leader_election.cc:288] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [CANDIDATE]: Term 1 election: Requested vote from peers 0d3acdc669f44938b3e20f4284d13d61 (127.0.98.1:36865), 83c5d539bbe34a1a9b421ee66a621b33 (127.0.98.3:34003)
I20260924 03:01:16.225876 19090 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "555d9770c2d04b5a8cd47fb30a081d80" candidate_uuid: "c798c769d59749778f6ef9df5321ce8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3acdc669f44938b3e20f4284d13d61"
I20260924 03:01:16.225875 19352 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "555d9770c2d04b5a8cd47fb30a081d80" candidate_uuid: "c798c769d59749778f6ef9df5321ce8c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "83c5d539bbe34a1a9b421ee66a621b33"
I20260924 03:01:16.225960 19352 raft_consensus.cc:3037] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:16.225960 19090 raft_consensus.cc:3037] T 555d9770c2d04b5a8cd47fb30a081d80 P 0d3acdc669f44938b3e20f4284d13d61 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:16.226858 19090 raft_consensus.cc:2454] T 555d9770c2d04b5a8cd47fb30a081d80 P 0d3acdc669f44938b3e20f4284d13d61 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c798c769d59749778f6ef9df5321ce8c in term 1.
I20260924 03:01:16.226855 19352 raft_consensus.cc:2454] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c798c769d59749778f6ef9df5321ce8c in term 1.
I20260924 03:01:16.227013 19156 leader_election.cc:302] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3acdc669f44938b3e20f4284d13d61, c798c769d59749778f6ef9df5321ce8c; no voters:
I20260924 03:01:16.227098 19420 raft_consensus.cc:2781] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:16.227259 19420 raft_consensus.cc:686] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [term 1 LEADER]: Becoming Leader. State: Replica: c798c769d59749778f6ef9df5321ce8c, State: Running, Role: LEADER
I20260924 03:01:16.227348 19420 consensus_queue.cc:237] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3acdc669f44938b3e20f4284d13d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36865 } } peers { permanent_uuid: "c798c769d59749778f6ef9df5321ce8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43355 } } peers { permanent_uuid: "83c5d539bbe34a1a9b421ee66a621b33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34003 } }
I20260924 03:01:16.228176 18948 catalog_manager.cc:5495] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c reported cstate change: term changed from 0 to 1, leader changed from <none> to c798c769d59749778f6ef9df5321ce8c (127.0.98.2). New cstate: current_term: 1 leader_uuid: "c798c769d59749778f6ef9df5321ce8c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3acdc669f44938b3e20f4284d13d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36865 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c798c769d59749778f6ef9df5321ce8c" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 43355 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "83c5d539bbe34a1a9b421ee66a621b33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34003 } health_report { overall_health: UNKNOWN } } }
I20260924 03:01:16.248209 19409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:16.245884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51898 (local address 127.0.98.2:43355)
0924 03:01:16.245931 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:16.245933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:16.245939 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:01:16.245980 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:16.245981 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:16.245999 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:16.246063 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:01:16.246169 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.246688 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.247525 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:16.247713 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:16.247783 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:16.247920 (+ 137us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:01:16.248010 (+ 90us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:01:16.248081 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:16.248121 (+ 40us) server_negotiation.cc:299] Negotiation successful
0924 03:01:16.248130 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260924 03:01:16.250036 19352 raft_consensus.cc:1261] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33 [term 1 FOLLOWER]: Refusing update from remote peer c798c769d59749778f6ef9df5321ce8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:01:16.250037 19090 raft_consensus.cc:1261] T 555d9770c2d04b5a8cd47fb30a081d80 P 0d3acdc669f44938b3e20f4284d13d61 [term 1 FOLLOWER]: Refusing update from remote peer c798c769d59749778f6ef9df5321ce8c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:01:16.250402 19422 consensus_queue.cc:1035] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "83c5d539bbe34a1a9b421ee66a621b33" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 34003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:01:16.250552 19420 consensus_queue.cc:1035] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d3acdc669f44938b3e20f4284d13d61" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 36865 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:01:16.253530 19426 mvcc.cc:205] Tried to move back new op lower bound from 7332736517117526016 to 7332736517027954688. Current Snapshot: MvccSnapshot[applied={T|T < 7332736517117526016 or (T in {7332736517117526016})}]
I20260924 03:01:16.253651 19427 mvcc.cc:205] Tried to move back new op lower bound from 7332736517117526016 to 7332736517027954688. Current Snapshot: MvccSnapshot[applied={T|T < 7332736517117526016}]
I20260924 03:01:16.255139 18948 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:01:16.255220 18948 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:01:16.255863 18948 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9b4aed1cc6ae40ed940600789126c5f6: Sending DeleteTablet for 3 replicas of tablet 555d9770c2d04b5a8cd47fb30a081d80
I20260924 03:01:16.256172 19332 tablet_service.cc:1514] Processing DeleteTablet for tablet 555d9770c2d04b5a8cd47fb30a081d80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:16 UTC) from {username='slave'} at 127.0.0.1:39476
I20260924 03:01:16.256178 19070 tablet_service.cc:1514] Processing DeleteTablet for tablet 555d9770c2d04b5a8cd47fb30a081d80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:16 UTC) from {username='slave'} at 127.0.0.1:36706
I20260924 03:01:16.256410 19201 tablet_service.cc:1514] Processing DeleteTablet for tablet 555d9770c2d04b5a8cd47fb30a081d80 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:16 UTC) from {username='slave'} at 127.0.0.1:51890
I20260924 03:01:16.256451 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 19008
I20260924 03:01:16.258618 19437 tablet_replica.cc:330] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c: stopping tablet replica
I20260924 03:01:16.258714 19437 raft_consensus.cc:2229] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:01:16.258809 19437 raft_consensus.cc:2258] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:01:16.259158 19437 ts_tablet_manager.cc:1905] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:01:16.260651 19437 ts_tablet_manager.cc:1918] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:01:16.260713 19437 log.cc:1196] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/555d9770c2d04b5a8cd47fb30a081d80
I20260924 03:01:16.260906 19437 ts_tablet_manager.cc:1939] T 555d9770c2d04b5a8cd47fb30a081d80 P c798c769d59749778f6ef9df5321ce8c: Deleting consensus metadata
I20260924 03:01:16.261232 18932 catalog_manager.cc:4841] TS c798c769d59749778f6ef9df5321ce8c (127.0.98.2:43355): tablet 555d9770c2d04b5a8cd47fb30a081d80 (table test-table [id=3c8f3ce95ffa40929e97819adfedc7a6]) successfully deleted
W20260924 03:01:16.262004 18934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:36865 (error 108)
I20260924 03:01:16.262158 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 19139
I20260924 03:01:16.263027 19435 tablet_replica.cc:330] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33: stopping tablet replica
I20260924 03:01:16.263144 19435 raft_consensus.cc:2229] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:01:16.263257 19435 raft_consensus.cc:2258] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:01:16.263660 19435 ts_tablet_manager.cc:1905] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:01:16.265142 19435 ts_tablet_manager.cc:1918] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:01:16.265213 19435 log.cc:1196] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/555d9770c2d04b5a8cd47fb30a081d80
I20260924 03:01:16.265451 19435 ts_tablet_manager.cc:1939] T 555d9770c2d04b5a8cd47fb30a081d80 P 83c5d539bbe34a1a9b421ee66a621b33: Deleting consensus metadata
I20260924 03:01:16.265838 18932 catalog_manager.cc:4841] TS 83c5d539bbe34a1a9b421ee66a621b33 (127.0.98.3:34003): tablet 555d9770c2d04b5a8cd47fb30a081d80 (table test-table [id=3c8f3ce95ffa40929e97819adfedc7a6]) successfully deleted
I20260924 03:01:16.266692 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 19270
I20260924 03:01:16.266840 19410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:16.266704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:36865 (local address 127.0.0.1:36710)
0924 03:01:16.266760 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:16.266809 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36865: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260924 03:01:16.270877 18934 catalog_manager.cc:4584] TS 0d3acdc669f44938b3e20f4284d13d61 (127.0.98.1:36865): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 555d9770c2d04b5a8cd47fb30a081d80: Network error: Client connection negotiation failed: client connection to 127.0.98.1:36865: connect: Connection refused (error 111)
I20260924 03:01:16.271072 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 18917
2026-09-24T03:01:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (663 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:16 dist-test-slave-sb6p krb5kdc[19441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:16 dist-test-slave-sb6p krb5kdc[19441](info): set up 2 sockets
Sep 24 03:01:16 dist-test-slave-sb6p krb5kdc[19441](info): commencing operation
krb5kdc: starting...
W20260924 03:01:17.222066 392 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:17 dist-test-slave-sb6p krb5kdc[19441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:17Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:01:17.329514 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33147
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:33147
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:17.407189 19454 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:17.407383 19454 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:17.407403 19454 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:17.408535 19454 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:17.408573 19454 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:17.408586 19454 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:17.408597 19454 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:17.408608 19454 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:17.409976 19454 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:33147
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:33147
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19454
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:17.410167 19454 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:17.410336 19454 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:17.414887 19459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:17.414893 19460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:17.414893 19462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:17.415052 19454 server_base.cc:1034] running on GCE node
I20260924 03:01:17.415323 19454 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:17.415629 19454 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:17.416769 19454 hybrid_clock.cc:648] HybridClock initialized: now 1790218877416752 us; error 28 us; skew 500 ppm
Sep 24 03:01:17 dist-test-slave-sb6p krb5kdc[19441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:17.419086 19454 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:01:17.419502 19454 webserver.cc:460] Webserver started at http://127.0.98.62:33909/ using document root <none> and password file <none>
I20260924 03:01:17.419695 19454 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:17.419739 19454 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:17.419854 19454 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:17.420851 19454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "86b23322fad04ef181225f4c0e7b962c"
format_stamp: "Formatted at 2026-09-24 03:01:17 on dist-test-slave-sb6p"
server_key: "86be225b3eb54bb83521c77a0fd78210"
server_key_iv: "d682b74c910897d4b41ff8d35af8f611"
server_key_version: "encryptionkey@0"
I20260924 03:01:17.421157 19454 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "86b23322fad04ef181225f4c0e7b962c"
format_stamp: "Formatted at 2026-09-24 03:01:17 on dist-test-slave-sb6p"
server_key: "86be225b3eb54bb83521c77a0fd78210"
server_key_iv: "d682b74c910897d4b41ff8d35af8f611"
server_key_version: "encryptionkey@0"
I20260924 03:01:17.422459 19454 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:01:17.423133 19469 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:17.423282 19454 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:17.423358 19454 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "86b23322fad04ef181225f4c0e7b962c"
format_stamp: "Formatted at 2026-09-24 03:01:17 on dist-test-slave-sb6p"
server_key: "86be225b3eb54bb83521c77a0fd78210"
server_key_iv: "d682b74c910897d4b41ff8d35af8f611"
server_key_version: "encryptionkey@0"
I20260924 03:01:17.423452 19454 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:17.431576 19454 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:17.433277 19454 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:17.433399 19454 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:17.437073 19454 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:33147
I20260924 03:01:17.437103 19521 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:33147 every 8 connection(s)
I20260924 03:01:17.437402 19454 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:17.438150 19522 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:17.440348 19522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c: Bootstrap starting.
I20260924 03:01:17.440894 19522 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:17.441143 19522 log.cc:824] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:17.441769 19522 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c: No bootstrap required, opened a new log
I20260924 03:01:17.443364 19522 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b23322fad04ef181225f4c0e7b962c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33147 } }
I20260924 03:01:17.443517 19522 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:17.443540 19522 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86b23322fad04ef181225f4c0e7b962c, State: Initialized, Role: FOLLOWER
I20260924 03:01:17.443677 19522 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b23322fad04ef181225f4c0e7b962c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33147 } }
I20260924 03:01:17.443743 19522 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:17.443773 19522 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:17.443758 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 19454
I20260924 03:01:17.443821 19522 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:17.443869 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:17.444060 392 external_mini_cluster.cc:1428] Setting key ac940871149f61921f0bed5025fda83a
I20260924 03:01:17.444355 19522 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b23322fad04ef181225f4c0e7b962c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33147 } }
I20260924 03:01:17.444483 19522 leader_election.cc:302] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86b23322fad04ef181225f4c0e7b962c; no voters:
I20260924 03:01:17.444653 19522 leader_election.cc:288] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:17.444670 19527 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:17.444826 19527 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [term 1 LEADER]: Becoming Leader. State: Replica: 86b23322fad04ef181225f4c0e7b962c, State: Running, Role: LEADER
I20260924 03:01:17.444891 19522 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:17.444908 19527 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b23322fad04ef181225f4c0e7b962c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33147 } }
I20260924 03:01:17.445976 19528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86b23322fad04ef181225f4c0e7b962c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b23322fad04ef181225f4c0e7b962c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33147 } } }
I20260924 03:01:17.446085 19528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:17.446342 19541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:17.446415 19529 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86b23322fad04ef181225f4c0e7b962c. Latest consensus state: current_term: 1 leader_uuid: "86b23322fad04ef181225f4c0e7b962c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86b23322fad04ef181225f4c0e7b962c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 33147 } } }
I20260924 03:01:17.446756 19529 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:17.447208 19541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:17.448611 19541 catalog_manager.cc:1347] Generated new cluster ID: 110a266ead524121af086c2c55bc3e39
I20260924 03:01:17.448661 19541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:01:17 dist-test-slave-sb6p krb5kdc[19441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218877, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:17.455404 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:17.444487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50722 (local address 127.0.98.62:33147)
0924 03:01:17.444669 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:17.444678 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:17.444710 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:01:17.444829 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:17.444836 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:17.444893 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:17.445045 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:17.445776 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.447334 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.448415 (+ 1081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.449002 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.449100 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:17.451882 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:17.451886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:17.451887 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:17.451899 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:17.453703 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:17.454080 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:17.454083 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:17.454083 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:17.454192 (+ 109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:17.454379 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:17.454380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:17.454380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:17.454686 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:17.454821 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:17.454946 (+ 125us) server_negotiation.cc:299] Negotiation successful
0924 03:01:17.455175 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":93,"threads_started":1}
I20260924 03:01:17.461692 19541 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:17.462234 19541 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:17.466136 19541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c: Generated new TSK 0
I20260924 03:01:17.466292 19541 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.98.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
Entry for principal kudu/127.0.98.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab.
I20260924 03:01:17.491465 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33147
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:17.573904 19548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:17.574095 19548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:17.574116 19548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:17.575316 19548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:17.575356 19548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:17.575384 19548 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:01:17.576855 19548 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kudu_127.0.98.1.keytab
--principal=kudu/127.0.98.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19548
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:17.577075 19548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:17.577283 19548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:17.581919 19556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:17.582003 19553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:17.582010 19554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:17.582091 19548 server_base.cc:1034] running on GCE node
I20260924 03:01:17.582355 19548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:17.582577 19548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:17.583710 19548 hybrid_clock.cc:648] HybridClock initialized: now 1790218877583699 us; error 28 us; skew 500 ppm
Sep 24 03:01:17 dist-test-slave-sb6p krb5kdc[19441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:17.586076 19548 init.cc:377] Logged in from keytab as kudu/127.0.98.1@KRBTEST.COM (short username kudu)
I20260924 03:01:17.586555 19548 webserver.cc:460] Webserver started at http://127.0.98.1:35085/ using document root <none> and password file <none>
I20260924 03:01:17.586766 19548 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:17.586808 19548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:17.586927 19548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:17.587944 19548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "85695b5bde4341aead9168e9b8f9d734"
format_stamp: "Formatted at 2026-09-24 03:01:17 on dist-test-slave-sb6p"
server_key: "2d2498e6ac85228e18c99748f2ba249b"
server_key_iv: "f4bd7f3111d2df28cde41d9b8a542471"
server_key_version: "encryptionkey@0"
I20260924 03:01:17.588248 19548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "85695b5bde4341aead9168e9b8f9d734"
format_stamp: "Formatted at 2026-09-24 03:01:17 on dist-test-slave-sb6p"
server_key: "2d2498e6ac85228e18c99748f2ba249b"
server_key_iv: "f4bd7f3111d2df28cde41d9b8a542471"
server_key_version: "encryptionkey@0"
I20260924 03:01:17.589542 19548 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:01:17.590207 19563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:17.590353 19548 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:17.590474 19548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "85695b5bde4341aead9168e9b8f9d734"
format_stamp: "Formatted at 2026-09-24 03:01:17 on dist-test-slave-sb6p"
server_key: "2d2498e6ac85228e18c99748f2ba249b"
server_key_iv: "f4bd7f3111d2df28cde41d9b8a542471"
server_key_version: "encryptionkey@0"
I20260924 03:01:17.590567 19548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:17.605227 19548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:17.607024 19548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:17.607143 19548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:17.607348 19548 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:17.607640 19548 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:17.607671 19548 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:17.607692 19548 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:17.607714 19548 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:17.612950 19548 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:45065
I20260924 03:01:17.613013 19676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:45065 every 8 connection(s)
I20260924 03:01:17.613276 19548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:01:17.615770 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 19548
I20260924 03:01:17.615880 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:01:17.616092 392 external_mini_cluster.cc:1428] Setting key 070eb2cc86af08a432e3bd62d8900eb1
Sep 24 03:01:17 dist-test-slave-sb6p krb5kdc[19441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.1@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:17.622514 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:17.613890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:59795 (local address 127.0.98.62:33147)
0924 03:01:17.613951 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:17.613953 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:17.614132 (+ 179us) server_negotiation.cc:407] Connection header received
0924 03:01:17.615082 (+ 950us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:17.615083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:17.615097 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:17.615137 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:17.615722 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.616411 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.617488 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.617695 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.617749 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:17.619999 (+ 2250us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:17.620001 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:17.620001 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:17.620007 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:17.621215 (+ 1208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:17.621665 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:17.621666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:17.621667 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:17.621712 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:17.621950 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:17.621950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:17.621951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:17.622041 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:17.622084 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:17.622383 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 03:01:17.622449 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260924 03:01:17.622793 19679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:17.613885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33147 (local address 127.0.98.1:59795)
0924 03:01:17.614020 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:17.614054 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:17.614984 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:17.615206 (+ 222us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:17.615209 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:17.615453 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:17.615644 (+ 191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:17.615648 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.616515 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.616516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:17.617403 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:17.617406 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.617761 (+ 355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.617762 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:17.617891 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:17.618354 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:17.618362 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:17.619914 (+ 1552us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:17.621292 (+ 1378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:17.621300 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:17.621302 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:17.621600 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:17.621768 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:17.621769 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:17.621769 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:17.621897 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:17.622096 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:17.622101 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:17.622260 (+ 159us) client_negotiation.cc:769] Sending connection context
0924 03:01:17.622375 (+ 115us) client_negotiation.cc:240] Negotiation successful
0924 03:01:17.622566 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":69,"threads_started":1}
I20260924 03:01:17.623312 19677 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33147
I20260924 03:01:17.623414 19677 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:17.623613 19677 heartbeater.cc:511] Master 127.0.98.62:33147 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.98.2@KRBTEST.COM; defaulting to no policy
I20260924 03:01:17.624116 19486 ts_manager.cc:194] Registered new tserver with Master: 85695b5bde4341aead9168e9b8f9d734 (127.0.98.1:45065)
I20260924 03:01:17.624732 19486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.1@KRBTEST.COM'} at 127.0.98.1:59795
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
Entry for principal kudu/127.0.98.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab.
I20260924 03:01:17.639542 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33147
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:17.717597 19682 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:17.717820 19682 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:17.717844 19682 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:17.719000 19682 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:17.719039 19682 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:17.719066 19682 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:01:17.720523 19682 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kudu_127.0.98.2.keytab
--principal=kudu/127.0.98.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19682
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:17.720743 19682 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:17.720954 19682 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:17.725719 19690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:17.725754 19687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:17.725731 19688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:17.725994 19682 server_base.cc:1034] running on GCE node
I20260924 03:01:17.726186 19682 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:17.726405 19682 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:17.727561 19682 hybrid_clock.cc:648] HybridClock initialized: now 1790218877727558 us; error 49 us; skew 500 ppm
Sep 24 03:01:17 dist-test-slave-sb6p krb5kdc[19441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:17.729941 19682 init.cc:377] Logged in from keytab as kudu/127.0.98.2@KRBTEST.COM (short username kudu)
I20260924 03:01:17.730374 19682 webserver.cc:460] Webserver started at http://127.0.98.2:38553/ using document root <none> and password file <none>
I20260924 03:01:17.730620 19682 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:17.730669 19682 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:17.730793 19682 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:17.731822 19682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "b18010edc4cd4d17a027fd8d6ff23a08"
format_stamp: "Formatted at 2026-09-24 03:01:17 on dist-test-slave-sb6p"
server_key: "51adc44b80a4584cd4df8abb5eb6f1ef"
server_key_iv: "ded222119c363dd8023beba9aa23b974"
server_key_version: "encryptionkey@0"
I20260924 03:01:17.732151 19682 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "b18010edc4cd4d17a027fd8d6ff23a08"
format_stamp: "Formatted at 2026-09-24 03:01:17 on dist-test-slave-sb6p"
server_key: "51adc44b80a4584cd4df8abb5eb6f1ef"
server_key_iv: "ded222119c363dd8023beba9aa23b974"
server_key_version: "encryptionkey@0"
I20260924 03:01:17.733317 19682 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:01:17.734007 19697 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:17.734156 19682 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:01:17.734236 19682 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "b18010edc4cd4d17a027fd8d6ff23a08"
format_stamp: "Formatted at 2026-09-24 03:01:17 on dist-test-slave-sb6p"
server_key: "51adc44b80a4584cd4df8abb5eb6f1ef"
server_key_iv: "ded222119c363dd8023beba9aa23b974"
server_key_version: "encryptionkey@0"
I20260924 03:01:17.734300 19682 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:17.770884 19682 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:17.772584 19682 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:17.772716 19682 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:17.772928 19682 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:17.773221 19682 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:17.773252 19682 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:17.773280 19682 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:17.773305 19682 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:17.778745 19682 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:41635
I20260924 03:01:17.778789 19810 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:41635 every 8 connection(s)
I20260924 03:01:17.779121 19682 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:01:17.784262 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 19682
I20260924 03:01:17.784391 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:01:17.784619 392 external_mini_cluster.cc:1428] Setting key 7b87ee61aa8e7266fef5a091749cdbc5
Sep 24 03:01:17 dist-test-slave-sb6p krb5kdc[19441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.2@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:17.788120 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:17.779771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:42597 (local address 127.0.98.62:33147)
0924 03:01:17.779847 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:17.779850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:17.780027 (+ 177us) server_negotiation.cc:407] Connection header received
0924 03:01:17.780955 (+ 928us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:17.780956 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:17.780966 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:17.781014 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:17.781590 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.782075 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.783024 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.783219 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.783271 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:17.785495 (+ 2224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:17.785496 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:17.785497 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:17.785504 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:17.786805 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:17.787231 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:17.787232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:17.787232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:17.787276 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:17.787545 (+ 269us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:17.787546 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:17.787546 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:17.787636 (+ 90us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:17.787679 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:17.787953 (+ 274us) server_negotiation.cc:299] Negotiation successful
0924 03:01:17.788035 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260924 03:01:17.788271 19813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:17.779730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33147 (local address 127.0.98.2:42597)
0924 03:01:17.779892 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:17.779929 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:17.780853 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:17.781078 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:17.781081 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:17.781334 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:17.781526 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:17.781531 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.782134 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.782135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:17.782953 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:17.782956 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.783281 (+ 325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.783281 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:17.783384 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:17.783867 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:17.783876 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:17.785408 (+ 1532us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:17.786874 (+ 1466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:17.786881 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:17.786883 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:17.787165 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:17.787322 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:17.787323 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:17.787323 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:17.787489 (+ 166us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:17.787685 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:17.787689 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:17.787820 (+ 131us) client_negotiation.cc:769] Sending connection context
0924 03:01:17.787913 (+ 93us) client_negotiation.cc:240] Negotiation successful
0924 03:01:17.788062 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":82,"threads_started":1}
I20260924 03:01:17.788825 19811 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33147
I20260924 03:01:17.788923 19811 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:17.789090 19811 heartbeater.cc:511] Master 127.0.98.62:33147 requested a full tablet report, sending...
I20260924 03:01:17.789402 19486 ts_manager.cc:194] Registered new tserver with Master: b18010edc4cd4d17a027fd8d6ff23a08 (127.0.98.2:41635)
I20260924 03:01:17.789856 19486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.2@KRBTEST.COM'} at 127.0.98.2:42597
WARNING: no policy specified for kudu/127.0.98.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
Entry for principal kudu/127.0.98.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab.
I20260924 03:01:17.814805 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33147
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:33809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:17.893211 19816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:17.893409 19816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:17.893440 19816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:17.894635 19816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:17.894675 19816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:17.894709 19816 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:01:17.896231 19816 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:33809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/krb5kdc/kudu_127.0.98.3.keytab
--principal=kudu/127.0.98.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:33147
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19816
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:17.896458 19816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:17.896658 19816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:17.901388 19824 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:17.901409 19821 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:17.901388 19822 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:17.901543 19816 server_base.cc:1034] running on GCE node
I20260924 03:01:17.901785 19816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:17.901966 19816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:17.903095 19816 hybrid_clock.cc:648] HybridClock initialized: now 1790218877903085 us; error 25 us; skew 500 ppm
Sep 24 03:01:17 dist-test-slave-sb6p krb5kdc[19441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:17.905310 19816 init.cc:377] Logged in from keytab as kudu/127.0.98.3@KRBTEST.COM (short username kudu)
I20260924 03:01:17.905674 19816 webserver.cc:460] Webserver started at http://127.0.98.3:34027/ using document root <none> and password file <none>
I20260924 03:01:17.905835 19816 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:17.905874 19816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:17.905967 19816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:17.906961 19816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "4766930959e140459bd3e4eec614846b"
format_stamp: "Formatted at 2026-09-24 03:01:17 on dist-test-slave-sb6p"
server_key: "136bd7d42b03e205eef2b9edc50ff94f"
server_key_iv: "38276609c78fef43a002389246c78149"
server_key_version: "encryptionkey@0"
I20260924 03:01:17.907235 19816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "4766930959e140459bd3e4eec614846b"
format_stamp: "Formatted at 2026-09-24 03:01:17 on dist-test-slave-sb6p"
server_key: "136bd7d42b03e205eef2b9edc50ff94f"
server_key_iv: "38276609c78fef43a002389246c78149"
server_key_version: "encryptionkey@0"
I20260924 03:01:17.908401 19816 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:01:17.909040 19831 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:17.909184 19816 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:17.909257 19816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "4766930959e140459bd3e4eec614846b"
format_stamp: "Formatted at 2026-09-24 03:01:17 on dist-test-slave-sb6p"
server_key: "136bd7d42b03e205eef2b9edc50ff94f"
server_key_iv: "38276609c78fef43a002389246c78149"
server_key_version: "encryptionkey@0"
I20260924 03:01:17.909309 19816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:17.920475 19816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:17.922240 19816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:17.922348 19816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:17.922582 19816 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:17.922860 19816 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:17.922891 19816 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:17.922921 19816 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:17.922938 19816 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:17.928174 19816 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:43951
I20260924 03:01:17.928236 19944 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:43951 every 8 connection(s)
I20260924 03:01:17.928510 19816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:01:17.928996 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 19816
I20260924 03:01:17.929103 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:01:17.929284 392 external_mini_cluster.cc:1428] Setting key 3941fdfe0129c82fc4d893c7ef25d365
Sep 24 03:01:17 dist-test-slave-sb6p krb5kdc[19441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.3@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:17.937011 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:17.929137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:53375 (local address 127.0.98.62:33147)
0924 03:01:17.929183 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:17.929185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:17.929391 (+ 206us) server_negotiation.cc:407] Connection header received
0924 03:01:17.930425 (+ 1034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:17.930426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:17.930437 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:17.930487 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:17.931110 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.931593 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.932563 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.932756 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.932810 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:17.934736 (+ 1926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:17.934737 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:17.934737 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:17.934744 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:17.935763 (+ 1019us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:17.936110 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:17.936111 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:17.936112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:17.936164 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:17.936364 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:17.936365 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:17.936365 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:17.936484 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:17.936539 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:17.936838 (+ 299us) server_negotiation.cc:299] Negotiation successful
0924 03:01:17.936923 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:01:17.937170 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:17.929130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:33147 (local address 127.0.98.3:53375)
0924 03:01:17.929279 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:17.929318 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:17.930332 (+ 1014us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:17.930578 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:17.930582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:17.930834 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:17.931022 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:17.931026 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.931671 (+ 645us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.931672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:17.932482 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:17.932484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.932822 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.932823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:17.932944 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:17.933378 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:17.933386 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:17.934658 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:17.935831 (+ 1173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:17.935837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:17.935839 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:17.936046 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:17.936215 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:17.936215 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:17.936216 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:17.936313 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:17.936547 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:17.936554 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:17.936679 (+ 125us) client_negotiation.cc:769] Sending connection context
0924 03:01:17.936815 (+ 136us) client_negotiation.cc:240] Negotiation successful
0924 03:01:17.936965 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":88,"threads_started":1}
I20260924 03:01:17.937561 19945 heartbeater.cc:347] Connected to a master server at 127.0.98.62:33147
I20260924 03:01:17.937644 19945 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:17.937810 19945 heartbeater.cc:511] Master 127.0.98.62:33147 requested a full tablet report, sending...
I20260924 03:01:17.938166 19486 ts_manager.cc:194] Registered new tserver with Master: 4766930959e140459bd3e4eec614846b (127.0.98.3:43951)
I20260924 03:01:17.938494 19486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.98.3@KRBTEST.COM'} at 127.0.98.3:53375
I20260924 03:01:17.940836 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:01:17.947382 19526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:17.942110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50728 (local address 127.0.98.62:33147)
0924 03:01:17.942157 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:17.942158 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:17.942254 (+ 96us) server_negotiation.cc:407] Connection header received
0924 03:01:17.942292 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:17.942293 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:17.942301 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:17.942347 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:17.942923 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.943383 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.944028 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.944195 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.944234 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:17.945285 (+ 1051us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:17.945286 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:17.945286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:17.945292 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:17.946406 (+ 1114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:17.946767 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:17.946768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:17.946769 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:17.946817 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:17.946973 (+ 156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:17.946973 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:17.946973 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:17.947095 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:17.947163 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:17.947237 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 03:01:17.947320 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:01:17.948727 19486 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50728:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:01:17.950297 19486 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:01:17.956466 19953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:17.952342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43951 (local address 127.0.0.1:33778)
0924 03:01:17.952848 (+ 506us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:17.952855 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:17.952895 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:17.953149 (+ 254us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:17.953150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:17.953151 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:17.953188 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:17.953189 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.954319 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.954320 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:17.955282 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:17.955285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.956147 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.956148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:17.956233 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:17.956248 (+ 15us) client_negotiation.cc:769] Sending connection context
0924 03:01:17.956321 (+ 73us) client_negotiation.cc:240] Negotiation successful
0924 03:01:17.956385 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"mutex_wait_us":4,"thread_start_us":41,"threads_started":1}
I20260924 03:01:17.956463 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:17.952240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45508 (local address 127.0.98.2:41635)
0924 03:01:17.952392 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:17.952397 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:17.952559 (+ 162us) server_negotiation.cc:407] Connection header received
0924 03:01:17.952639 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:17.952641 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:17.952655 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:17.952726 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:17.952910 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.953967 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.955356 (+ 1389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.956065 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.956137 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:17.956196 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:17.956341 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 03:01:17.956382 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260924 03:01:17.956707 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:17.952583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33778 (local address 127.0.98.3:43951)
0924 03:01:17.952717 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:17.952721 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:17.952956 (+ 235us) server_negotiation.cc:407] Connection header received
0924 03:01:17.952996 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:17.952998 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:17.953013 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:17.953075 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:17.953260 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.954238 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.955369 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.956063 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.956137 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:17.956196 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:17.956594 (+ 398us) server_negotiation.cc:299] Negotiation successful
0924 03:01:17.956635 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
I20260924 03:01:17.956861 19954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:17.952137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41635 (local address 127.0.0.1:45508)
0924 03:01:17.952450 (+ 313us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:17.952492 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:17.952565 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:17.952741 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:17.952745 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:17.952747 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:17.952811 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:17.952813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.954063 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.954064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:17.955288 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:17.955291 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.956148 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.956150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:17.956246 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:17.956250 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:01:17.956321 (+ 71us) client_negotiation.cc:240] Negotiation successful
0924 03:01:17.956385 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":291,"mutex_wait_us":27,"thread_start_us":66,"threads_started":1}
I20260924 03:01:17.957374 19879 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2740ce9822419f85f60815d44b0b0f (DEFAULT_TABLE table=test-table [id=c9190e8191bc42149e8b6f42a19a6315]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:17.957504 19745 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2740ce9822419f85f60815d44b0b0f (DEFAULT_TABLE table=test-table [id=c9190e8191bc42149e8b6f42a19a6315]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:17.957679 19879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2740ce9822419f85f60815d44b0b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:17.957805 19745 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2740ce9822419f85f60815d44b0b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:17.958817 19956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:17.952158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45065 (local address 127.0.0.1:41082)
0924 03:01:17.952726 (+ 568us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:17.952734 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:17.952784 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:17.953017 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:17.953018 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:17.953020 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:17.953061 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:17.953063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.953941 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.953941 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:17.955290 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:17.955292 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.958588 (+ 3296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.958589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:17.958651 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:17.958655 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:01:17.958709 (+ 54us) client_negotiation.cc:240] Negotiation successful
0924 03:01:17.958751 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"mutex_wait_us":43,"thread_start_us":51,"threads_started":1}
I20260924 03:01:17.958881 19957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:17.952419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41082 (local address 127.0.98.1:45065)
0924 03:01:17.952566 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:17.952570 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:17.952781 (+ 211us) server_negotiation.cc:407] Connection header received
0924 03:01:17.952848 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:17.952851 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:17.952868 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:17.952934 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:17.953129 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.953832 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.956232 (+ 2400us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:17.956926 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:17.957293 (+ 367us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:17.957357 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:17.958755 (+ 1398us) server_negotiation.cc:299] Negotiation successful
0924 03:01:17.958804 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260924 03:01:17.959801 19611 tablet_service.cc:1467] Processing CreateTablet for tablet 4c2740ce9822419f85f60815d44b0b0f (DEFAULT_TABLE table=test-table [id=c9190e8191bc42149e8b6f42a19a6315]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:17.960110 19611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c2740ce9822419f85f60815d44b0b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:17.960408 19959 tablet_bootstrap.cc:492] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b: Bootstrap starting.
I20260924 03:01:17.960704 19960 tablet_bootstrap.cc:492] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08: Bootstrap starting.
I20260924 03:01:17.961145 19959 tablet_bootstrap.cc:654] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:17.961229 19960 tablet_bootstrap.cc:654] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:17.961691 19959 log.cc:824] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:17.961719 19960 log.cc:824] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:17.962389 19959 tablet_bootstrap.cc:492] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b: No bootstrap required, opened a new log
I20260924 03:01:17.962456 19959 ts_tablet_manager.cc:1397] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:01:17.962598 19960 tablet_bootstrap.cc:492] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08: No bootstrap required, opened a new log
I20260924 03:01:17.962661 19960 ts_tablet_manager.cc:1397] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:01:17.962860 19963 tablet_bootstrap.cc:492] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734: Bootstrap starting.
I20260924 03:01:17.963486 19963 tablet_bootstrap.cc:654] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:17.964016 19963 log.cc:824] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:17.964078 19960 raft_consensus.cc:348] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18010edc4cd4d17a027fd8d6ff23a08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41635 } } peers { permanent_uuid: "85695b5bde4341aead9168e9b8f9d734" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } } peers { permanent_uuid: "4766930959e140459bd3e4eec614846b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43951 } }
I20260924 03:01:17.964263 19960 raft_consensus.cc:374] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:17.964308 19960 raft_consensus.cc:729] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b18010edc4cd4d17a027fd8d6ff23a08, State: Initialized, Role: FOLLOWER
I20260924 03:01:17.964356 19959 raft_consensus.cc:348] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18010edc4cd4d17a027fd8d6ff23a08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41635 } } peers { permanent_uuid: "85695b5bde4341aead9168e9b8f9d734" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } } peers { permanent_uuid: "4766930959e140459bd3e4eec614846b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43951 } }
I20260924 03:01:17.964439 19960 consensus_queue.cc:260] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18010edc4cd4d17a027fd8d6ff23a08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41635 } } peers { permanent_uuid: "85695b5bde4341aead9168e9b8f9d734" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } } peers { permanent_uuid: "4766930959e140459bd3e4eec614846b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43951 } }
I20260924 03:01:17.964532 19959 raft_consensus.cc:374] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:17.964568 19959 raft_consensus.cc:729] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4766930959e140459bd3e4eec614846b, State: Initialized, Role: FOLLOWER
I20260924 03:01:17.964674 19959 consensus_queue.cc:260] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18010edc4cd4d17a027fd8d6ff23a08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41635 } } peers { permanent_uuid: "85695b5bde4341aead9168e9b8f9d734" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } } peers { permanent_uuid: "4766930959e140459bd3e4eec614846b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43951 } }
I20260924 03:01:17.964709 19960 ts_tablet_manager.cc:1428] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:01:17.964754 19963 tablet_bootstrap.cc:492] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734: No bootstrap required, opened a new log
I20260924 03:01:17.964818 19963 ts_tablet_manager.cc:1397] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260924 03:01:17.964820 19811 heartbeater.cc:503] Master 127.0.98.62:33147 was elected leader, sending a full tablet report...
I20260924 03:01:17.964931 19959 ts_tablet_manager.cc:1428] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260924 03:01:17.964974 19945 heartbeater.cc:503] Master 127.0.98.62:33147 was elected leader, sending a full tablet report...
I20260924 03:01:17.966647 19963 raft_consensus.cc:348] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18010edc4cd4d17a027fd8d6ff23a08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41635 } } peers { permanent_uuid: "85695b5bde4341aead9168e9b8f9d734" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } } peers { permanent_uuid: "4766930959e140459bd3e4eec614846b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43951 } }
I20260924 03:01:17.966778 19963 raft_consensus.cc:374] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:17.966810 19963 raft_consensus.cc:729] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85695b5bde4341aead9168e9b8f9d734, State: Initialized, Role: FOLLOWER
I20260924 03:01:17.966938 19963 consensus_queue.cc:260] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18010edc4cd4d17a027fd8d6ff23a08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41635 } } peers { permanent_uuid: "85695b5bde4341aead9168e9b8f9d734" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } } peers { permanent_uuid: "4766930959e140459bd3e4eec614846b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43951 } }
I20260924 03:01:17.967165 19677 heartbeater.cc:503] Master 127.0.98.62:33147 was elected leader, sending a full tablet report...
I20260924 03:01:17.967170 19963 ts_tablet_manager.cc:1428] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20260924 03:01:18.029670 19812 tablet.cc:2369] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:01:18.113965 19678 tablet.cc:2369] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:01:18.179064 19946 tablet.cc:2369] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:01:18.208367 19966 raft_consensus.cc:482] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:01:18.208480 19966 raft_consensus.cc:504] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18010edc4cd4d17a027fd8d6ff23a08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41635 } } peers { permanent_uuid: "85695b5bde4341aead9168e9b8f9d734" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } } peers { permanent_uuid: "4766930959e140459bd3e4eec614846b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43951 } }
I20260924 03:01:18.208854 19966 leader_election.cc:288] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b18010edc4cd4d17a027fd8d6ff23a08 (127.0.98.2:41635), 85695b5bde4341aead9168e9b8f9d734 (127.0.98.1:45065)
I20260924 03:01:18.209122 19965 raft_consensus.cc:482] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:01:18.209226 19965 raft_consensus.cc:504] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18010edc4cd4d17a027fd8d6ff23a08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41635 } } peers { permanent_uuid: "85695b5bde4341aead9168e9b8f9d734" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } } peers { permanent_uuid: "4766930959e140459bd3e4eec614846b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43951 } }
I20260924 03:01:18.209710 19965 leader_election.cc:288] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 85695b5bde4341aead9168e9b8f9d734 (127.0.98.1:45065), 4766930959e140459bd3e4eec614846b (127.0.98.3:43951)
I20260924 03:01:18.212347 19947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.209068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:41635 (local address 127.0.98.3:33847)
0924 03:01:18.209213 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:18.209224 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:18.209277 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:18.209437 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:18.209438 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:18.209440 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:18.209507 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.209509 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.210320 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.210321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.211429 (+ 1108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.211432 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.212118 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.212120 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.212200 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.212203 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:18.212242 (+ 39us) client_negotiation.cc:240] Negotiation successful
0924 03:01:18.212274 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"mutex_wait_us":1}
I20260924 03:01:18.212582 19955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.209130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:33847 (local address 127.0.98.2:41635)
0924 03:01:18.209175 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:18.209177 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:18.209327 (+ 150us) server_negotiation.cc:407] Connection header received
0924 03:01:18.209354 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:18.209355 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:18.209366 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:18.209416 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:18.209576 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.210243 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.211514 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.212019 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.212080 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.212090 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:18.212497 (+ 407us) server_negotiation.cc:299] Negotiation successful
0924 03:01:18.212529 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:01:18.212802 19765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2740ce9822419f85f60815d44b0b0f" candidate_uuid: "4766930959e140459bd3e4eec614846b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b18010edc4cd4d17a027fd8d6ff23a08" is_pre_election: true
I20260924 03:01:18.212898 19765 raft_consensus.cc:2454] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4766930959e140459bd3e4eec614846b in term 0.
I20260924 03:01:18.213109 19834 leader_election.cc:302] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4766930959e140459bd3e4eec614846b, b18010edc4cd4d17a027fd8d6ff23a08; no voters:
I20260924 03:01:18.213140 19957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.209170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34507 (local address 127.0.98.1:45065)
0924 03:01:18.209308 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:18.209311 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:18.209343 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:01:18.209408 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:18.209409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:18.209419 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:18.209464 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:18.209676 (+ 212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.210348 (+ 672us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.212284 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.212771 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.212812 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.212820 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:18.213040 (+ 220us) server_negotiation.cc:299] Negotiation successful
0924 03:01:18.213071 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":114}
I20260924 03:01:18.213245 19966 raft_consensus.cc:2781] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:01:18.213296 19966 raft_consensus.cc:482] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:01:18.213340 19966 raft_consensus.cc:3037] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:18.213415 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.209069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45065 (local address 127.0.98.3:34507)
0924 03:01:18.209279 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:18.209289 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:18.209349 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:18.209512 (+ 163us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:18.209513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:18.209515 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:18.209582 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.209584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.210425 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.210426 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.212184 (+ 1758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.212187 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.212885 (+ 698us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.212886 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.212953 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.212956 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:18.213248 (+ 292us) client_negotiation.cc:240] Negotiation successful
0924 03:01:18.213285 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":196,"mutex_wait_us":61,"thread_start_us":87,"threads_started":1}
I20260924 03:01:18.213474 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.210082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:52447 (local address 127.0.98.3:43951)
0924 03:01:18.210114 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:18.210116 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:18.210207 (+ 91us) server_negotiation.cc:407] Connection header received
0924 03:01:18.210281 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:18.210282 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:18.210288 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:18.210311 (+ 23us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:18.210489 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.211099 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.212629 (+ 1530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.213100 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.213139 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.213146 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:18.213354 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 03:01:18.213387 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":18}
I20260924 03:01:18.213681 19813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.209650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45065 (local address 127.0.98.2:55565)
0924 03:01:18.209818 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:18.209827 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:18.209871 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:18.210021 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:18.210022 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:18.210024 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:18.210071 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.210073 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.210962 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.210962 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.212122 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.212125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.213428 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.213430 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.213519 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.213523 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:01:18.213551 (+ 28us) client_negotiation.cc:240] Negotiation successful
0924 03:01:18.213590 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151}
I20260924 03:01:18.213685 19970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.209933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:43951 (local address 127.0.98.2:52447)
0924 03:01:18.210157 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:18.210166 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:18.210226 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:18.210358 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:18.210358 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:18.210360 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:18.210421 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.210422 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.211193 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.211194 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.212548 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.212551 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.213212 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.213213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.213279 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.213282 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:18.213437 (+ 155us) client_negotiation.cc:240] Negotiation successful
0924 03:01:18.213561 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":13,"thread_start_us":76,"threads_started":1}
I20260924 03:01:18.213717 19631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2740ce9822419f85f60815d44b0b0f" candidate_uuid: "4766930959e140459bd3e4eec614846b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85695b5bde4341aead9168e9b8f9d734" is_pre_election: true
I20260924 03:01:18.213848 19631 raft_consensus.cc:2454] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4766930959e140459bd3e4eec614846b in term 0.
I20260924 03:01:18.213973 19899 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2740ce9822419f85f60815d44b0b0f" candidate_uuid: "b18010edc4cd4d17a027fd8d6ff23a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4766930959e140459bd3e4eec614846b" is_pre_election: true
I20260924 03:01:18.214258 19969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.209705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:55565 (local address 127.0.98.1:45065)
0924 03:01:18.209817 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:18.209819 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:18.209917 (+ 98us) server_negotiation.cc:407] Connection header received
0924 03:01:18.209938 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:18.209939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:18.209953 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:18.210120 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:18.210146 (+ 26us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.210878 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.212231 (+ 1353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.213157 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.214127 (+ 970us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.214138 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:18.214154 (+ 16us) server_negotiation.cc:299] Negotiation successful
0924 03:01:18.214196 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":65,"threads_started":1}
I20260924 03:01:18.214233 19966 raft_consensus.cc:504] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18010edc4cd4d17a027fd8d6ff23a08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41635 } } peers { permanent_uuid: "85695b5bde4341aead9168e9b8f9d734" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } } peers { permanent_uuid: "4766930959e140459bd3e4eec614846b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43951 } }
I20260924 03:01:18.214325 19899 raft_consensus.cc:2379] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate b18010edc4cd4d17a027fd8d6ff23a08 in current term 1: Already voted for candidate 4766930959e140459bd3e4eec614846b in this term.
I20260924 03:01:18.214356 19966 leader_election.cc:288] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [CANDIDATE]: Term 1 election: Requested vote from peers b18010edc4cd4d17a027fd8d6ff23a08 (127.0.98.2:41635), 85695b5bde4341aead9168e9b8f9d734 (127.0.98.1:45065)
I20260924 03:01:18.214406 19631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2740ce9822419f85f60815d44b0b0f" candidate_uuid: "b18010edc4cd4d17a027fd8d6ff23a08" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85695b5bde4341aead9168e9b8f9d734" is_pre_election: true
I20260924 03:01:18.214468 19631 raft_consensus.cc:2454] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b18010edc4cd4d17a027fd8d6ff23a08 in term 0.
I20260924 03:01:18.214588 19765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2740ce9822419f85f60815d44b0b0f" candidate_uuid: "4766930959e140459bd3e4eec614846b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b18010edc4cd4d17a027fd8d6ff23a08"
I20260924 03:01:18.214637 19631 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c2740ce9822419f85f60815d44b0b0f" candidate_uuid: "4766930959e140459bd3e4eec614846b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "85695b5bde4341aead9168e9b8f9d734"
I20260924 03:01:18.214643 19698 leader_election.cc:302] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 85695b5bde4341aead9168e9b8f9d734, b18010edc4cd4d17a027fd8d6ff23a08; no voters: 4766930959e140459bd3e4eec614846b
I20260924 03:01:18.214665 19765 raft_consensus.cc:3037] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:18.214702 19631 raft_consensus.cc:3037] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:18.215292 19765 raft_consensus.cc:2454] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4766930959e140459bd3e4eec614846b in term 1.
I20260924 03:01:18.215385 19965 raft_consensus.cc:2741] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260924 03:01:18.215430 19631 raft_consensus.cc:2454] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4766930959e140459bd3e4eec614846b in term 1.
I20260924 03:01:18.215448 19834 leader_election.cc:302] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4766930959e140459bd3e4eec614846b, b18010edc4cd4d17a027fd8d6ff23a08; no voters:
I20260924 03:01:18.215520 19966 raft_consensus.cc:2781] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:18.215656 19966 raft_consensus.cc:686] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [term 1 LEADER]: Becoming Leader. State: Replica: 4766930959e140459bd3e4eec614846b, State: Running, Role: LEADER
I20260924 03:01:18.215729 19966 consensus_queue.cc:237] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18010edc4cd4d17a027fd8d6ff23a08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41635 } } peers { permanent_uuid: "85695b5bde4341aead9168e9b8f9d734" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } } peers { permanent_uuid: "4766930959e140459bd3e4eec614846b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43951 } }
I20260924 03:01:18.216548 19486 catalog_manager.cc:5495] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b reported cstate change: term changed from 0 to 1, leader changed from <none> to 4766930959e140459bd3e4eec614846b (127.0.98.3). New cstate: current_term: 1 leader_uuid: "4766930959e140459bd3e4eec614846b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b18010edc4cd4d17a027fd8d6ff23a08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "85695b5bde4341aead9168e9b8f9d734" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4766930959e140459bd3e4eec614846b" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 43951 } health_report { overall_health: HEALTHY } } }
I20260924 03:01:18.239162 19958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.236932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33786 (local address 127.0.98.3:43951)
0924 03:01:18.236972 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:18.236974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:18.236981 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:01:18.237022 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:18.237022 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:18.237041 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:18.237099 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:01:18.237220 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.237718 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.238405 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.238646 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.238699 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.238871 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:01:18.238978 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:01:18.239049 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:18.239069 (+ 20us) server_negotiation.cc:299] Negotiation successful
0924 03:01:18.239088 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:01:18.240819 19765 raft_consensus.cc:1261] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08 [term 1 FOLLOWER]: Refusing update from remote peer 4766930959e140459bd3e4eec614846b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:01:18.240835 19631 raft_consensus.cc:1261] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734 [term 1 FOLLOWER]: Refusing update from remote peer 4766930959e140459bd3e4eec614846b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:01:18.241112 19966 consensus_queue.cc:1035] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [LEADER]: Connected to new peer: Peer: permanent_uuid: "b18010edc4cd4d17a027fd8d6ff23a08" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 41635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:01:18.241266 19971 consensus_queue.cc:1035] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [LEADER]: Connected to new peer: Peer: permanent_uuid: "85695b5bde4341aead9168e9b8f9d734" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 45065 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:01:18.244201 19975 mvcc.cc:205] Tried to move back new op lower bound from 7332736525271826432 to 7332736525172129792. Current Snapshot: MvccSnapshot[applied={T|T < 7332736525271826432 or (T in {7332736525271826432})}]
I20260924 03:01:18.245354 19976 mvcc.cc:205] Tried to move back new op lower bound from 7332736525271826432 to 7332736525172129792. Current Snapshot: MvccSnapshot[applied={T|T < 7332736525271826432 or (T in {7332736525271826432})}]
I20260924 03:01:18.245607 19485 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:01:18.245703 19485 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50728:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:01:18.246381 19485 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 86b23322fad04ef181225f4c0e7b962c: Sending DeleteTablet for 3 replicas of tablet 4c2740ce9822419f85f60815d44b0b0f
I20260924 03:01:18.246719 19611 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c2740ce9822419f85f60815d44b0b0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:41082
I20260924 03:01:18.246769 19745 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c2740ce9822419f85f60815d44b0b0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:45508
I20260924 03:01:18.246948 19879 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c2740ce9822419f85f60815d44b0b0f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:18 UTC) from {username='kudu', principal='kudu/127.0.98.62@KRBTEST.COM'} at 127.0.0.1:33778
I20260924 03:01:18.246974 19983 tablet_replica.cc:330] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734: stopping tablet replica
I20260924 03:01:18.247073 19983 raft_consensus.cc:2229] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:01:18.247153 19985 tablet_replica.cc:330] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b: stopping tablet replica
I20260924 03:01:18.247201 19983 raft_consensus.cc:2258] T 4c2740ce9822419f85f60815d44b0b0f P 85695b5bde4341aead9168e9b8f9d734 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:01:18.247243 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 19548
I20260924 03:01:18.247251 19985 raft_consensus.cc:2229] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:01:18.247332 19985 raft_consensus.cc:2258] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:01:18.247566 19984 tablet_replica.cc:330] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08: stopping tablet replica
I20260924 03:01:18.247643 19984 raft_consensus.cc:2229] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:01:18.247722 19984 raft_consensus.cc:2258] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:01:18.247713 19985 ts_tablet_manager.cc:1905] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:01:18.248153 19984 ts_tablet_manager.cc:1905] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:01:18.249466 19985 ts_tablet_manager.cc:1918] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:01:18.249524 19985 log.cc:1196] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/4c2740ce9822419f85f60815d44b0b0f
I20260924 03:01:18.249765 19985 ts_tablet_manager.cc:1939] T 4c2740ce9822419f85f60815d44b0b0f P 4766930959e140459bd3e4eec614846b: Deleting consensus metadata
I20260924 03:01:18.249787 19984 ts_tablet_manager.cc:1918] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:01:18.249858 19984 log.cc:1196] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790218647334776-392-0/minicluster-data/ts-1/wal/wals/4c2740ce9822419f85f60815d44b0b0f
I20260924 03:01:18.250083 19984 ts_tablet_manager.cc:1939] T 4c2740ce9822419f85f60815d44b0b0f P b18010edc4cd4d17a027fd8d6ff23a08: Deleting consensus metadata
I20260924 03:01:18.250118 19472 catalog_manager.cc:4841] TS 4766930959e140459bd3e4eec614846b (127.0.98.3:43951): tablet 4c2740ce9822419f85f60815d44b0b0f (table test-table [id=c9190e8191bc42149e8b6f42a19a6315]) successfully deleted
I20260924 03:01:18.250419 19472 catalog_manager.cc:4841] TS b18010edc4cd4d17a027fd8d6ff23a08 (127.0.98.2:41635): tablet 4c2740ce9822419f85f60815d44b0b0f (table test-table [id=c9190e8191bc42149e8b6f42a19a6315]) successfully deleted
W20260924 03:01:18.253712 19470 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:45065 (error 108)
I20260924 03:01:18.253947 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 19682
I20260924 03:01:18.254485 19956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.254089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:45065 (local address 127.0.0.1:41096)
0924 03:01:18.254407 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:18.254449 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45065: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300}
W20260924 03:01:18.254817 19470 catalog_manager.cc:4584] TS 85695b5bde4341aead9168e9b8f9d734 (127.0.98.1:45065): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c2740ce9822419f85f60815d44b0b0f: Network error: Client connection negotiation failed: client connection to 127.0.98.1:45065: connect: Connection refused (error 111)
I20260924 03:01:18.258733 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 19816
I20260924 03:01:18.262893 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 19454
2026-09-24T03:01:18Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1992 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-24T03:01:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:18Z Disabled control of system clock
I20260924 03:01:18.287557 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44925
--webserver_interface=127.0.98.62
--webserver_port=0
--builtin_ntp_servers=127.0.98.20:44645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:44925
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:01:18.365289 19990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:18.365506 19990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:18.365526 19990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:18.366626 19990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:18.366657 19990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:18.366669 19990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:18.366680 19990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:18.366690 19990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:18.367962 19990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:44925
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:44925
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:18.368149 19990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:18.368404 19990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:18.372969 19998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:18.373035 19990 server_base.cc:1034] running on GCE node
W20260924 03:01:18.372965 19995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:18.372948 19996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:18.373430 19990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:18.373689 19990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:18.374816 19990 hybrid_clock.cc:648] HybridClock initialized: now 1790218878374801 us; error 33 us; skew 500 ppm
I20260924 03:01:18.376080 19990 webserver.cc:460] Webserver started at http://127.0.98.62:40617/ using document root <none> and password file <none>
I20260924 03:01:18.376312 19990 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:18.376382 19990 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:18.376513 19990 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:18.377553 19990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "8c6e49ed4fe3491da3f0170af337eac2"
format_stamp: "Formatted at 2026-09-24 03:01:18 on dist-test-slave-sb6p"
server_key: "10f4b4417cfb998af2f081c9e6359a4b"
server_key_iv: "de159f033b7910ed6cc961b759c3001b"
server_key_version: "encryptionkey@0"
I20260924 03:01:18.377863 19990 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "8c6e49ed4fe3491da3f0170af337eac2"
format_stamp: "Formatted at 2026-09-24 03:01:18 on dist-test-slave-sb6p"
server_key: "10f4b4417cfb998af2f081c9e6359a4b"
server_key_iv: "de159f033b7910ed6cc961b759c3001b"
server_key_version: "encryptionkey@0"
I20260924 03:01:18.379159 19990 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:01:18.379849 20004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:18.379997 19990 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:01:18.380075 19990 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "8c6e49ed4fe3491da3f0170af337eac2"
format_stamp: "Formatted at 2026-09-24 03:01:18 on dist-test-slave-sb6p"
server_key: "10f4b4417cfb998af2f081c9e6359a4b"
server_key_iv: "de159f033b7910ed6cc961b759c3001b"
server_key_version: "encryptionkey@0"
I20260924 03:01:18.380153 19990 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:18.406576 19990 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:18.406842 19990 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:18.406940 19990 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:18.410599 19990 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:44925
I20260924 03:01:18.410665 20056 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:44925 every 8 connection(s)
I20260924 03:01:18.410987 19990 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:18.411501 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 19990
I20260924 03:01:18.411634 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:18.411832 392 external_mini_cluster.cc:1428] Setting key 3ade9e6b56d1b3a0d8daabe3cc1fb061
I20260924 03:01:18.412071 20057 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:18.415109 20057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2: Bootstrap starting.
I20260924 03:01:18.415690 20057 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:18.415939 20057 log.cc:824] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:18.416551 20057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2: No bootstrap required, opened a new log
I20260924 03:01:18.417161 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.412361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43674 (local address 127.0.98.62:44925)
0924 03:01:18.412553 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:18.412560 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:18.412584 (+ 24us) server_negotiation.cc:407] Connection header received
0924 03:01:18.413076 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:18.413083 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:18.413145 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:18.413255 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:18.413468 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.414312 (+ 844us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.415369 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.415798 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.415920 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.416564 (+ 644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:18.416568 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:18.416569 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:18.416744 (+ 175us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:18.416777 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:18.416787 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:18.416841 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:18.416908 (+ 67us) server_negotiation.cc:299] Negotiation successful
0924 03:01:18.416986 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
I20260924 03:01:18.418473 20057 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6e49ed4fe3491da3f0170af337eac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44925 } }
I20260924 03:01:18.418637 20057 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:18.418681 20057 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c6e49ed4fe3491da3f0170af337eac2, State: Initialized, Role: FOLLOWER
I20260924 03:01:18.418792 20057 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6e49ed4fe3491da3f0170af337eac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44925 } }
I20260924 03:01:18.418850 20057 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:18.418879 20057 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:18.418926 20057 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:18.419459 20057 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6e49ed4fe3491da3f0170af337eac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44925 } }
I20260924 03:01:18.419544 20057 leader_election.cc:302] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c6e49ed4fe3491da3f0170af337eac2; no voters:
I20260924 03:01:18.419699 20057 leader_election.cc:288] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:18.419739 20062 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:18.419864 20062 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [term 1 LEADER]: Becoming Leader. State: Replica: 8c6e49ed4fe3491da3f0170af337eac2, State: Running, Role: LEADER
I20260924 03:01:18.419932 20057 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:18.419970 20062 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6e49ed4fe3491da3f0170af337eac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44925 } }
I20260924 03:01:18.420255 20062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c6e49ed4fe3491da3f0170af337eac2. Latest consensus state: current_term: 1 leader_uuid: "8c6e49ed4fe3491da3f0170af337eac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6e49ed4fe3491da3f0170af337eac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44925 } } }
I20260924 03:01:18.420338 20062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:18.420272 20064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c6e49ed4fe3491da3f0170af337eac2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c6e49ed4fe3491da3f0170af337eac2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 44925 } } }
I20260924 03:01:18.420476 20064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:18.420884 20068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:18.421343 20068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:18.422760 20068 catalog_manager.cc:1347] Generated new cluster ID: 969629bd3f4d49b59a5ff77110dd7467
I20260924 03:01:18.422812 20068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:18.438357 20068 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:18.438951 20068 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:18.444806 20068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2: Generated new TSK 0
I20260924 03:01:18.445003 20068 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:18.450359 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.1:0
--local_ip_for_outbound_sockets=127.0.98.1
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44925
--builtin_ntp_servers=127.0.98.20:44645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260924 03:01:18.529273 20081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:18.529490 20081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:18.529512 20081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:18.530699 20081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:18.530737 20081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:18.530762 20081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.1
I20260924 03:01:18.532073 20081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.98.1
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20081
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.1
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:18.532302 20081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:18.532604 20081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:18.538221 20089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:18.538196 20087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:18.538363 20086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:18.538393 20081 server_base.cc:1034] running on GCE node
I20260924 03:01:18.538650 20081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:18.538883 20081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:18.540030 20081 hybrid_clock.cc:648] HybridClock initialized: now 1790218878540016 us; error 29 us; skew 500 ppm
I20260924 03:01:18.541155 20081 webserver.cc:460] Webserver started at http://127.0.98.1:44919/ using document root <none> and password file <none>
I20260924 03:01:18.541371 20081 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:18.541422 20081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:18.541543 20081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:18.542585 20081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-0/data/instance:
uuid: "4922eb076b7d4011a2d4cc13a12e480d"
format_stamp: "Formatted at 2026-09-24 03:01:18 on dist-test-slave-sb6p"
server_key: "b24e1a5a77ef96bf49b247ec2c0411e3"
server_key_iv: "655081637442623a2a7bab08e1bee35d"
server_key_version: "encryptionkey@0"
I20260924 03:01:18.542891 20081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-0/wal/instance:
uuid: "4922eb076b7d4011a2d4cc13a12e480d"
format_stamp: "Formatted at 2026-09-24 03:01:18 on dist-test-slave-sb6p"
server_key: "b24e1a5a77ef96bf49b247ec2c0411e3"
server_key_iv: "655081637442623a2a7bab08e1bee35d"
server_key_version: "encryptionkey@0"
I20260924 03:01:18.544073 20081 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:01:18.544736 20095 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:18.544904 20081 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:01:18.544983 20081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-0/wal
uuid: "4922eb076b7d4011a2d4cc13a12e480d"
format_stamp: "Formatted at 2026-09-24 03:01:18 on dist-test-slave-sb6p"
server_key: "b24e1a5a77ef96bf49b247ec2c0411e3"
server_key_iv: "655081637442623a2a7bab08e1bee35d"
server_key_version: "encryptionkey@0"
I20260924 03:01:18.545039 20081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:18.558218 20081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:18.558461 20081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:18.558604 20081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:18.558781 20081 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:18.559036 20081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:18.559064 20081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:18.559089 20081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:18.559105 20081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:18.564162 20081 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.1:44131
I20260924 03:01:18.564270 20208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.1:44131 every 8 connection(s)
I20260924 03:01:18.564514 20081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-0/data/info.pb
I20260924 03:01:18.564795 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 20081
I20260924 03:01:18.564904 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-0/wal/instance
I20260924 03:01:18.565114 392 external_mini_cluster.cc:1428] Setting key 986430705dc5bc9563986dc6062e3bc9
I20260924 03:01:18.566056 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.2:0
--local_ip_for_outbound_sockets=127.0.98.2
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44925
--builtin_ntp_servers=127.0.98.20:44645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:01:18.570277 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.565254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.1:57173 (local address 127.0.98.62:44925)
0924 03:01:18.565306 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:18.565308 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:18.565585 (+ 277us) server_negotiation.cc:407] Connection header received
0924 03:01:18.566278 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:18.566279 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:18.566293 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:18.566343 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:18.567187 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.567671 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.568826 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.569107 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.569175 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.569986 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:18.569988 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:18.569988 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:18.569993 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:18.570022 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:18.570027 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:18.570088 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:18.570151 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 03:01:18.570191 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:01:18.570402 20211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.565268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44925 (local address 127.0.98.1:57173)
0924 03:01:18.565459 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:18.565501 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:18.566140 (+ 639us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:18.566419 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:18.566423 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:18.566434 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:18.567107 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.567115 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.567756 (+ 641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.567757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.568740 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.568742 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.569174 (+ 432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.569175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.569287 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.569856 (+ 569us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:01:18.569857 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:18.569885 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:01:18.569907 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:18.570088 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:18.570090 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:18.570093 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:18.570137 (+ 44us) client_negotiation.cc:240] Negotiation successful
0924 03:01:18.570224 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":84,"threads_started":1}
I20260924 03:01:18.570981 20209 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44925
I20260924 03:01:18.571062 20209 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:18.571226 20209 heartbeater.cc:511] Master 127.0.98.62:44925 requested a full tablet report, sending...
I20260924 03:01:18.571722 20021 ts_manager.cc:194] Registered new tserver with Master: 4922eb076b7d4011a2d4cc13a12e480d (127.0.98.1:44131)
I20260924 03:01:18.572640 20021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.1:57173
W20260924 03:01:18.645218 20212 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:18.645413 20212 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:18.645433 20212 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:18.646492 20212 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:18.646538 20212 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:18.646561 20212 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.2
I20260924 03:01:18.647907 20212 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.98.2
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20212
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.2
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:18.648095 20212 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:18.648401 20212 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:18.653293 20217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:18.653264 20220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:18.653301 20212 server_base.cc:1034] running on GCE node
W20260924 03:01:18.653270 20218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:18.653643 20212 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:18.653887 20212 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:18.655026 20212 hybrid_clock.cc:648] HybridClock initialized: now 1790218878655004 us; error 32 us; skew 500 ppm
I20260924 03:01:18.656222 20212 webserver.cc:460] Webserver started at http://127.0.98.2:45663/ using document root <none> and password file <none>
I20260924 03:01:18.656446 20212 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:18.656494 20212 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:18.656618 20212 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:18.657775 20212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-1/data/instance:
uuid: "0bc32e635bc7434ea4ffa342e70027a5"
format_stamp: "Formatted at 2026-09-24 03:01:18 on dist-test-slave-sb6p"
server_key: "c47a26d63779d9c0a6a927104cf93b9f"
server_key_iv: "8114eb0bb5ab30d6e5c4b1d92c2f5296"
server_key_version: "encryptionkey@0"
I20260924 03:01:18.658082 20212 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-1/wal/instance:
uuid: "0bc32e635bc7434ea4ffa342e70027a5"
format_stamp: "Formatted at 2026-09-24 03:01:18 on dist-test-slave-sb6p"
server_key: "c47a26d63779d9c0a6a927104cf93b9f"
server_key_iv: "8114eb0bb5ab30d6e5c4b1d92c2f5296"
server_key_version: "encryptionkey@0"
I20260924 03:01:18.659296 20212 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:01:18.659963 20226 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:18.660102 20212 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:01:18.660178 20212 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-1/wal
uuid: "0bc32e635bc7434ea4ffa342e70027a5"
format_stamp: "Formatted at 2026-09-24 03:01:18 on dist-test-slave-sb6p"
server_key: "c47a26d63779d9c0a6a927104cf93b9f"
server_key_iv: "8114eb0bb5ab30d6e5c4b1d92c2f5296"
server_key_version: "encryptionkey@0"
I20260924 03:01:18.660234 20212 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:18.669620 20212 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:18.669854 20212 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:18.669968 20212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:18.670145 20212 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:18.670423 20212 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:18.670454 20212 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:18.670480 20212 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:18.670504 20212 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:18.675805 20212 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.2:45679
I20260924 03:01:18.675849 20339 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.2:45679 every 8 connection(s)
I20260924 03:01:18.676143 20212 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-1/data/info.pb
I20260924 03:01:18.680840 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 20212
I20260924 03:01:18.680953 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-1/wal/instance
I20260924 03:01:18.681149 392 external_mini_cluster.cc:1428] Setting key ee500cfc1d53f3ea8c830d3a66d311b5
I20260924 03:01:18.681180 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.676811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.2:46343 (local address 127.0.98.62:44925)
0924 03:01:18.676859 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:18.676861 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:18.677092 (+ 231us) server_negotiation.cc:407] Connection header received
0924 03:01:18.677562 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:18.677563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:18.677572 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:18.677622 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:18.678293 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.678840 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.679824 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.680038 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.680101 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.680863 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:18.680864 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:18.680864 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:18.680876 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:18.680894 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:18.680899 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:18.680958 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:18.681065 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 03:01:18.681095 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:01:18.681368 20342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.676797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44925 (local address 127.0.98.2:46343)
0924 03:01:18.676980 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:18.677013 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:18.677480 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:18.677693 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:18.677696 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:18.677704 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:18.678221 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.678225 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.678910 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.678911 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.679729 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.679732 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.680107 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.680107 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.680209 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.680722 (+ 513us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:01:18.680722 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:18.680759 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:01:18.680782 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:18.680987 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:18.680993 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:18.680996 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:18.681063 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 03:01:18.681163 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":83,"threads_started":1}
I20260924 03:01:18.681826 20340 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44925
I20260924 03:01:18.681911 20340 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:18.682080 20340 heartbeater.cc:511] Master 127.0.98.62:44925 requested a full tablet report, sending...
I20260924 03:01:18.682091 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.98.3:0
--local_ip_for_outbound_sockets=127.0.98.3
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44925
--builtin_ntp_servers=127.0.98.20:44645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260924 03:01:18.682432 20021 ts_manager.cc:194] Registered new tserver with Master: 0bc32e635bc7434ea4ffa342e70027a5 (127.0.98.2:45679)
I20260924 03:01:18.682909 20021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.2:46343
W20260924 03:01:18.762013 20343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:18.762215 20343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:18.762244 20343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:18.763449 20343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:18.763495 20343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:18.763530 20343 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.98.3
I20260924 03:01:18.764894 20343 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.98.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.98.3
--webserver_port=0
--tserver_master_addrs=127.0.98.62:44925
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.98.3
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:18.765126 20343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:18.765394 20343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:18.770802 20348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:18.770787 20351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:18.770789 20349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:18.771289 20343 server_base.cc:1034] running on GCE node
I20260924 03:01:18.771492 20343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:18.771713 20343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:18.772848 20343 hybrid_clock.cc:648] HybridClock initialized: now 1790218878772833 us; error 28 us; skew 500 ppm
I20260924 03:01:18.773958 20343 webserver.cc:460] Webserver started at http://127.0.98.3:37107/ using document root <none> and password file <none>
I20260924 03:01:18.774180 20343 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:18.774228 20343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:18.774351 20343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:18.775493 20343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/data/instance:
uuid: "ee1ed8abe6b94cda9a149f4dfba28b97"
format_stamp: "Formatted at 2026-09-24 03:01:18 on dist-test-slave-sb6p"
server_key: "6c1fa5192b9e5ae367f13f62604218fb"
server_key_iv: "91b31326659c6421787c51f75e935af9"
server_key_version: "encryptionkey@0"
I20260924 03:01:18.775825 20343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/wal/instance:
uuid: "ee1ed8abe6b94cda9a149f4dfba28b97"
format_stamp: "Formatted at 2026-09-24 03:01:18 on dist-test-slave-sb6p"
server_key: "6c1fa5192b9e5ae367f13f62604218fb"
server_key_iv: "91b31326659c6421787c51f75e935af9"
server_key_version: "encryptionkey@0"
I20260924 03:01:18.777024 20343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:01:18.777681 20357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:18.777827 20343 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:18.777904 20343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/wal
uuid: "ee1ed8abe6b94cda9a149f4dfba28b97"
format_stamp: "Formatted at 2026-09-24 03:01:18 on dist-test-slave-sb6p"
server_key: "6c1fa5192b9e5ae367f13f62604218fb"
server_key_iv: "91b31326659c6421787c51f75e935af9"
server_key_version: "encryptionkey@0"
I20260924 03:01:18.777976 20343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:18.790275 20343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:18.790586 20343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:18.790714 20343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:18.790920 20343 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260924 03:01:18.791242 20343 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260924 03:01:18.791275 20343 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:18.791303 20343 ts_tablet_manager.cc:610] Registered 0 tablets
I20260924 03:01:18.791320 20343 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:18.796653 20343 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.3:44871
I20260924 03:01:18.796718 20470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.3:44871 every 8 connection(s)
I20260924 03:01:18.796998 20343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/data/info.pb
I20260924 03:01:18.801856 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.797631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:34243 (local address 127.0.98.62:44925)
0924 03:01:18.797713 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:18.797715 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:18.797943 (+ 228us) server_negotiation.cc:407] Connection header received
0924 03:01:18.798406 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:18.798407 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:18.798418 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:18.798468 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:18.799203 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.799694 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.800629 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.800813 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.800871 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.801645 (+ 774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:18.801646 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:18.801647 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:18.801650 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:18.801666 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:18.801670 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:18.801721 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:18.801774 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 03:01:18.801794 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260924 03:01:18.802080 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.797616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:44925 (local address 127.0.98.3:34243)
0924 03:01:18.797790 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:18.797833 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:18.798325 (+ 492us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:18.798551 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:18.798555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:18.798565 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:18.799064 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.799068 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.799765 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.799766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.800550 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.800552 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.800897 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.800898 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.801003 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.801499 (+ 496us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0924 03:01:18.801499 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:18.801526 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0924 03:01:18.801547 (+ 21us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:18.801726 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:18.801728 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:18.801731 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:18.801774 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 03:01:18.801848 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":82,"threads_started":1}
I20260924 03:01:18.802561 20471 heartbeater.cc:347] Connected to a master server at 127.0.98.62:44925
I20260924 03:01:18.802649 20471 heartbeater.cc:464] Registering TS with master...
I20260924 03:01:18.802814 20471 heartbeater.cc:511] Master 127.0.98.62:44925 requested a full tablet report, sending...
I20260924 03:01:18.803140 20021 ts_manager.cc:194] Registered new tserver with Master: ee1ed8abe6b94cda9a149f4dfba28b97 (127.0.98.3:44871)
I20260924 03:01:18.803491 20021 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.98.3:34243
I20260924 03:01:18.806954 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 20343
I20260924 03:01:18.807044 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/wal/instance
I20260924 03:01:18.807221 392 external_mini_cluster.cc:1428] Setting key 46358f3301b470c94ddb15484a6832d1
I20260924 03:01:18.808120 392 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260924 03:01:18.812033 20060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.809392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43680 (local address 127.0.98.62:44925)
0924 03:01:18.809440 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:18.809442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:18.809555 (+ 113us) server_negotiation.cc:407] Connection header received
0924 03:01:18.809599 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:18.809600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:18.809608 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:18.809651 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:18.809769 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.810228 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.810982 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.811156 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.811201 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.811867 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:18.811868 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:18.811868 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:18.811871 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:18.811880 (+ 9us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:18.811883 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:18.811911 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:18.811949 (+ 38us) server_negotiation.cc:299] Negotiation successful
0924 03:01:18.811964 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:01:18.813412 20021 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43680:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260924 03:01:18.814990 20021 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260924 03:01:18.820900 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.816903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44131 (local address 127.0.0.1:39446)
0924 03:01:18.817215 (+ 312us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:18.817225 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:18.817275 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:18.817557 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:18.817557 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:18.817560 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:18.817619 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.817621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.818864 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.818864 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.819948 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.819950 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.820569 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.820570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.820682 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.820698 (+ 16us) client_negotiation.cc:769] Sending connection context
0924 03:01:18.820767 (+ 69us) client_negotiation.cc:240] Negotiation successful
0924 03:01:18.820818 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":294,"mutex_wait_us":9,"thread_start_us":52,"threads_started":1}
I20260924 03:01:18.820919 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.817239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39446 (local address 127.0.98.1:44131)
0924 03:01:18.817413 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:18.817417 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:18.817426 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:01:18.817457 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:18.817459 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:18.817472 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:18.817551 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:18.817703 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.818787 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.820025 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.820493 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.820554 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.820704 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:18.820803 (+ 99us) server_negotiation.cc:299] Negotiation successful
0924 03:01:18.820843 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":76,"threads_started":1}
I20260924 03:01:18.821175 20479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.816890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45679 (local address 127.0.0.1:55076)
0924 03:01:18.817052 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:18.817088 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:18.817150 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:18.817514 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:18.817517 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:18.817519 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:18.817607 (+ 88us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.817610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.818645 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.818646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.819919 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.819922 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.820835 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.820836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.820895 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.820898 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:18.820927 (+ 29us) client_negotiation.cc:240] Negotiation successful
0924 03:01:18.821100 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260924 03:01:18.821300 20480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.817090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.3:44871 (local address 127.0.0.1:47818)
0924 03:01:18.817639 (+ 549us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:18.817649 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:18.817692 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:18.817923 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:18.817924 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:18.817926 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:18.817959 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.817961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.819105 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.819105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.819953 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:18.819955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.820961 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.820962 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:18.821016 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.821020 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:01:18.821066 (+ 46us) client_negotiation.cc:240] Negotiation successful
0924 03:01:18.821101 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"mutex_wait_us":2,"thread_start_us":79,"threads_started":1}
I20260924 03:01:18.821501 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.816971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55076 (local address 127.0.98.2:45679)
0924 03:01:18.817369 (+ 398us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:18.817373 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:18.817383 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:01:18.817414 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:18.817416 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:18.817428 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:18.817486 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:18.817697 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.818571 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.820009 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.820728 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.821144 (+ 416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.821304 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:18.821386 (+ 82us) server_negotiation.cc:299] Negotiation successful
0924 03:01:18.821424 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":109,"threads_started":1}
I20260924 03:01:18.821512 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:18.817175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47818 (local address 127.0.98.3:44871)
0924 03:01:18.817334 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:18.817338 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:18.817741 (+ 403us) server_negotiation.cc:407] Connection header received
0924 03:01:18.817780 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:18.817782 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:18.817796 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:18.817855 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:18.818038 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.819042 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.820032 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:18.820801 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:18.820851 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:18.820932 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:18.821382 (+ 450us) server_negotiation.cc:299] Negotiation successful
0924 03:01:18.821422 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260924 03:01:18.821609 20143 tablet_service.cc:1467] Processing CreateTablet for tablet 108fcbf5c84448b49466100f7190f4ed (DEFAULT_TABLE table=test-table [id=7c86d64a861e47d4ae774c7da309661c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:18.821971 20143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108fcbf5c84448b49466100f7190f4ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:18.822171 20274 tablet_service.cc:1467] Processing CreateTablet for tablet 108fcbf5c84448b49466100f7190f4ed (DEFAULT_TABLE table=test-table [id=7c86d64a861e47d4ae774c7da309661c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:18.822439 20274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108fcbf5c84448b49466100f7190f4ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:18.823148 20405 tablet_service.cc:1467] Processing CreateTablet for tablet 108fcbf5c84448b49466100f7190f4ed (DEFAULT_TABLE table=test-table [id=7c86d64a861e47d4ae774c7da309661c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260924 03:01:18.823433 20405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 108fcbf5c84448b49466100f7190f4ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:18.824499 20485 tablet_bootstrap.cc:492] T 108fcbf5c84448b49466100f7190f4ed P 4922eb076b7d4011a2d4cc13a12e480d: Bootstrap starting.
I20260924 03:01:18.825176 20485 tablet_bootstrap.cc:654] T 108fcbf5c84448b49466100f7190f4ed P 4922eb076b7d4011a2d4cc13a12e480d: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:18.825407 20485 log.cc:824] T 108fcbf5c84448b49466100f7190f4ed P 4922eb076b7d4011a2d4cc13a12e480d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:18.825569 20486 tablet_bootstrap.cc:492] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5: Bootstrap starting.
I20260924 03:01:18.826079 20486 tablet_bootstrap.cc:654] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:18.826135 20485 tablet_bootstrap.cc:492] T 108fcbf5c84448b49466100f7190f4ed P 4922eb076b7d4011a2d4cc13a12e480d: No bootstrap required, opened a new log
I20260924 03:01:18.826191 20485 ts_tablet_manager.cc:1397] T 108fcbf5c84448b49466100f7190f4ed P 4922eb076b7d4011a2d4cc13a12e480d: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:01:18.826308 20486 log.cc:824] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:18.826579 20488 tablet_bootstrap.cc:492] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97: Bootstrap starting.
I20260924 03:01:18.826980 20486 tablet_bootstrap.cc:492] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5: No bootstrap required, opened a new log
I20260924 03:01:18.827044 20486 ts_tablet_manager.cc:1397] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:01:18.827154 20488 tablet_bootstrap.cc:654] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:18.827445 20488 log.cc:824] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:18.828138 20488 tablet_bootstrap.cc:492] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97: No bootstrap required, opened a new log
I20260924 03:01:18.828203 20488 ts_tablet_manager.cc:1397] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260924 03:01:18.828370 20485 raft_consensus.cc:348] T 108fcbf5c84448b49466100f7190f4ed P 4922eb076b7d4011a2d4cc13a12e480d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4922eb076b7d4011a2d4cc13a12e480d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44131 } } peers { permanent_uuid: "ee1ed8abe6b94cda9a149f4dfba28b97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44871 } } peers { permanent_uuid: "0bc32e635bc7434ea4ffa342e70027a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45679 } }
I20260924 03:01:18.828500 20485 raft_consensus.cc:374] T 108fcbf5c84448b49466100f7190f4ed P 4922eb076b7d4011a2d4cc13a12e480d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:18.828532 20485 raft_consensus.cc:729] T 108fcbf5c84448b49466100f7190f4ed P 4922eb076b7d4011a2d4cc13a12e480d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4922eb076b7d4011a2d4cc13a12e480d, State: Initialized, Role: FOLLOWER
I20260924 03:01:18.828641 20485 consensus_queue.cc:260] T 108fcbf5c84448b49466100f7190f4ed P 4922eb076b7d4011a2d4cc13a12e480d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4922eb076b7d4011a2d4cc13a12e480d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44131 } } peers { permanent_uuid: "ee1ed8abe6b94cda9a149f4dfba28b97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44871 } } peers { permanent_uuid: "0bc32e635bc7434ea4ffa342e70027a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45679 } }
I20260924 03:01:18.828871 20485 ts_tablet_manager.cc:1428] T 108fcbf5c84448b49466100f7190f4ed P 4922eb076b7d4011a2d4cc13a12e480d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260924 03:01:18.829010 20209 heartbeater.cc:503] Master 127.0.98.62:44925 was elected leader, sending a full tablet report...
I20260924 03:01:18.829448 20486 raft_consensus.cc:348] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4922eb076b7d4011a2d4cc13a12e480d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44131 } } peers { permanent_uuid: "ee1ed8abe6b94cda9a149f4dfba28b97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44871 } } peers { permanent_uuid: "0bc32e635bc7434ea4ffa342e70027a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45679 } }
I20260924 03:01:18.829602 20486 raft_consensus.cc:374] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:18.829643 20486 raft_consensus.cc:729] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0bc32e635bc7434ea4ffa342e70027a5, State: Initialized, Role: FOLLOWER
I20260924 03:01:18.829751 20486 consensus_queue.cc:260] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4922eb076b7d4011a2d4cc13a12e480d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44131 } } peers { permanent_uuid: "ee1ed8abe6b94cda9a149f4dfba28b97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44871 } } peers { permanent_uuid: "0bc32e635bc7434ea4ffa342e70027a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45679 } }
I20260924 03:01:18.830008 20486 ts_tablet_manager.cc:1428] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:01:18.830026 20340 heartbeater.cc:503] Master 127.0.98.62:44925 was elected leader, sending a full tablet report...
I20260924 03:01:18.830544 20488 raft_consensus.cc:348] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4922eb076b7d4011a2d4cc13a12e480d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44131 } } peers { permanent_uuid: "ee1ed8abe6b94cda9a149f4dfba28b97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44871 } } peers { permanent_uuid: "0bc32e635bc7434ea4ffa342e70027a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45679 } }
I20260924 03:01:18.830677 20488 raft_consensus.cc:374] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:18.830708 20488 raft_consensus.cc:729] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee1ed8abe6b94cda9a149f4dfba28b97, State: Initialized, Role: FOLLOWER
I20260924 03:01:18.830822 20488 consensus_queue.cc:260] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4922eb076b7d4011a2d4cc13a12e480d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44131 } } peers { permanent_uuid: "ee1ed8abe6b94cda9a149f4dfba28b97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44871 } } peers { permanent_uuid: "0bc32e635bc7434ea4ffa342e70027a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45679 } }
I20260924 03:01:18.831064 20488 ts_tablet_manager.cc:1428] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260924 03:01:18.831107 20471 heartbeater.cc:503] Master 127.0.98.62:44925 was elected leader, sending a full tablet report...
W20260924 03:01:18.926657 20341 tablet.cc:2369] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:01:19.047524 20472 tablet.cc:2369] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260924 03:01:19.065165 20210 tablet.cc:2369] T 108fcbf5c84448b49466100f7190f4ed P 4922eb076b7d4011a2d4cc13a12e480d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260924 03:01:19.200851 20493 raft_consensus.cc:482] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260924 03:01:19.200997 20493 raft_consensus.cc:504] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4922eb076b7d4011a2d4cc13a12e480d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44131 } } peers { permanent_uuid: "ee1ed8abe6b94cda9a149f4dfba28b97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44871 } } peers { permanent_uuid: "0bc32e635bc7434ea4ffa342e70027a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45679 } }
I20260924 03:01:19.201300 20493 leader_election.cc:288] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4922eb076b7d4011a2d4cc13a12e480d (127.0.98.1:44131), 0bc32e635bc7434ea4ffa342e70027a5 (127.0.98.2:45679)
I20260924 03:01:19.204363 20484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:19.201545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:50645 (local address 127.0.98.1:44131)
0924 03:01:19.201596 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:19.201598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:19.201824 (+ 226us) server_negotiation.cc:407] Connection header received
0924 03:01:19.201888 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:19.201895 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:19.201904 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:19.201962 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:19.202130 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:19.202627 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:19.203703 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:19.204057 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:19.204117 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:19.204125 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:19.204261 (+ 136us) server_negotiation.cc:299] Negotiation successful
0924 03:01:19.204296 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:01:19.204370 20473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:19.201483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45679 (local address 127.0.98.3:36745)
0924 03:01:19.201569 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:19.201580 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:19.201625 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:19.201799 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:19.201800 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:19.201801 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:19.201879 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:19.201881 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:19.202775 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:19.202775 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:19.203588 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:19.203590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:19.204117 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:19.204118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:19.204188 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:19.204192 (+ 4us) client_negotiation.cc:769] Sending connection context
0924 03:01:19.204233 (+ 41us) client_negotiation.cc:240] Negotiation successful
0924 03:01:19.204279 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":53}
I20260924 03:01:19.204380 20494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:19.201483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44131 (local address 127.0.98.3:50645)
0924 03:01:19.201753 (+ 270us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:19.201764 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:19.201822 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:19.201976 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:19.201977 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:19.201979 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0924 03:01:19.202044 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:19.202045 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:19.202709 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:19.202710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:19.203603 (+ 893us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:19.203605 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:19.204134 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:19.204135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:19.204199 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:19.204202 (+ 3us) client_negotiation.cc:769] Sending connection context
0924 03:01:19.204250 (+ 48us) client_negotiation.cc:240] Negotiation successful
0924 03:01:19.204286 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":253,"mutex_wait_us":131,"thread_start_us":95,"threads_started":1}
I20260924 03:01:19.204371 20481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:19.201562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.98.3:36745 (local address 127.0.98.2:45679)
0924 03:01:19.201606 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:19.201608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:19.201618 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:01:19.201712 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:19.201712 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:19.201722 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:19.201795 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0924 03:01:19.201951 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:19.202695 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:19.203676 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:19.204045 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:19.204105 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:19.204117 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:19.204254 (+ 137us) server_negotiation.cc:299] Negotiation successful
0924 03:01:19.204291 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:01:19.204766 20163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108fcbf5c84448b49466100f7190f4ed" candidate_uuid: "ee1ed8abe6b94cda9a149f4dfba28b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4922eb076b7d4011a2d4cc13a12e480d" is_pre_election: true
I20260924 03:01:19.204818 20294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108fcbf5c84448b49466100f7190f4ed" candidate_uuid: "ee1ed8abe6b94cda9a149f4dfba28b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc32e635bc7434ea4ffa342e70027a5" is_pre_election: true
I20260924 03:01:19.204931 20163 raft_consensus.cc:2454] T 108fcbf5c84448b49466100f7190f4ed P 4922eb076b7d4011a2d4cc13a12e480d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee1ed8abe6b94cda9a149f4dfba28b97 in term 0.
I20260924 03:01:19.204949 20294 raft_consensus.cc:2454] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee1ed8abe6b94cda9a149f4dfba28b97 in term 0.
I20260924 03:01:19.205150 20360 leader_election.cc:302] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4922eb076b7d4011a2d4cc13a12e480d, ee1ed8abe6b94cda9a149f4dfba28b97; no voters:
I20260924 03:01:19.205277 20493 raft_consensus.cc:2781] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260924 03:01:19.205331 20493 raft_consensus.cc:482] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260924 03:01:19.205346 20493 raft_consensus.cc:3037] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:19.205993 20493 raft_consensus.cc:504] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4922eb076b7d4011a2d4cc13a12e480d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44131 } } peers { permanent_uuid: "ee1ed8abe6b94cda9a149f4dfba28b97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44871 } } peers { permanent_uuid: "0bc32e635bc7434ea4ffa342e70027a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45679 } }
I20260924 03:01:19.206127 20493 leader_election.cc:288] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [CANDIDATE]: Term 1 election: Requested vote from peers 4922eb076b7d4011a2d4cc13a12e480d (127.0.98.1:44131), 0bc32e635bc7434ea4ffa342e70027a5 (127.0.98.2:45679)
I20260924 03:01:19.206276 20163 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108fcbf5c84448b49466100f7190f4ed" candidate_uuid: "ee1ed8abe6b94cda9a149f4dfba28b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4922eb076b7d4011a2d4cc13a12e480d"
I20260924 03:01:19.206357 20163 raft_consensus.cc:3037] T 108fcbf5c84448b49466100f7190f4ed P 4922eb076b7d4011a2d4cc13a12e480d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:19.206420 20294 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "108fcbf5c84448b49466100f7190f4ed" candidate_uuid: "ee1ed8abe6b94cda9a149f4dfba28b97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0bc32e635bc7434ea4ffa342e70027a5"
I20260924 03:01:19.206575 20294 raft_consensus.cc:3037] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:19.207146 20163 raft_consensus.cc:2454] T 108fcbf5c84448b49466100f7190f4ed P 4922eb076b7d4011a2d4cc13a12e480d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee1ed8abe6b94cda9a149f4dfba28b97 in term 1.
I20260924 03:01:19.207261 20294 raft_consensus.cc:2454] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee1ed8abe6b94cda9a149f4dfba28b97 in term 1.
I20260924 03:01:19.207423 20359 leader_election.cc:302] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0bc32e635bc7434ea4ffa342e70027a5, ee1ed8abe6b94cda9a149f4dfba28b97; no voters:
I20260924 03:01:19.207540 20493 raft_consensus.cc:2781] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:19.207700 20493 raft_consensus.cc:686] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [term 1 LEADER]: Becoming Leader. State: Replica: ee1ed8abe6b94cda9a149f4dfba28b97, State: Running, Role: LEADER
I20260924 03:01:19.207785 20493 consensus_queue.cc:237] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4922eb076b7d4011a2d4cc13a12e480d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44131 } } peers { permanent_uuid: "ee1ed8abe6b94cda9a149f4dfba28b97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44871 } } peers { permanent_uuid: "0bc32e635bc7434ea4ffa342e70027a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45679 } }
I20260924 03:01:19.208577 20020 catalog_manager.cc:5495] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 reported cstate change: term changed from 0 to 1, leader changed from <none> to ee1ed8abe6b94cda9a149f4dfba28b97 (127.0.98.3). New cstate: current_term: 1 leader_uuid: "ee1ed8abe6b94cda9a149f4dfba28b97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4922eb076b7d4011a2d4cc13a12e480d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44131 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ee1ed8abe6b94cda9a149f4dfba28b97" member_type: VOTER last_known_addr { host: "127.0.98.3" port: 44871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0bc32e635bc7434ea4ffa342e70027a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45679 } health_report { overall_health: UNKNOWN } } }
I20260924 03:01:19.260918 20483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:19.258750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47826 (local address 127.0.98.3:44871)
0924 03:01:19.258810 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:19.258812 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:19.258819 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:01:19.258857 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:19.258858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:19.258876 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:19.258932 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:01:19.259052 (+ 120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:19.259515 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:19.260228 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:19.260414 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:19.260465 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:19.260610 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:01:19.260710 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:01:19.260779 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:19.260798 (+ 19us) server_negotiation.cc:299] Negotiation successful
0924 03:01:19.260816 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 03:01:19.262734 20294 raft_consensus.cc:1261] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5 [term 1 FOLLOWER]: Refusing update from remote peer ee1ed8abe6b94cda9a149f4dfba28b97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:01:19.262748 20163 raft_consensus.cc:1261] T 108fcbf5c84448b49466100f7190f4ed P 4922eb076b7d4011a2d4cc13a12e480d [term 1 FOLLOWER]: Refusing update from remote peer ee1ed8abe6b94cda9a149f4dfba28b97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:01:19.263077 20495 consensus_queue.cc:1035] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0bc32e635bc7434ea4ffa342e70027a5" member_type: VOTER last_known_addr { host: "127.0.98.2" port: 45679 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:01:19.263243 20493 consensus_queue.cc:1035] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4922eb076b7d4011a2d4cc13a12e480d" member_type: VOTER last_known_addr { host: "127.0.98.1" port: 44131 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:01:19.265378 20498 mvcc.cc:205] Tried to move back new op lower bound from 7332736529457274880 to 7332736529235558400. Current Snapshot: MvccSnapshot[applied={T|T < 7332736529457274880}]
I20260924 03:01:19.265923 20500 mvcc.cc:205] Tried to move back new op lower bound from 7332736529457274880 to 7332736529235558400. Current Snapshot: MvccSnapshot[applied={T|T < 7332736529457274880}]
I20260924 03:01:19.266098 20499 mvcc.cc:205] Tried to move back new op lower bound from 7332736529457274880 to 7332736529235558400. Current Snapshot: MvccSnapshot[applied={T|T < 7332736529457274880}]
I20260924 03:01:19.267812 20021 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:01:19.267907 20021 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43680:
table { table_name: "test-table" } modify_external_catalogs: true
I20260924 03:01:19.268613 20021 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8c6e49ed4fe3491da3f0170af337eac2: Sending DeleteTablet for 3 replicas of tablet 108fcbf5c84448b49466100f7190f4ed
I20260924 03:01:19.268898 20143 tablet_service.cc:1514] Processing DeleteTablet for tablet 108fcbf5c84448b49466100f7190f4ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:19 UTC) from {username='slave'} at 127.0.0.1:39446
I20260924 03:01:19.268922 20274 tablet_service.cc:1514] Processing DeleteTablet for tablet 108fcbf5c84448b49466100f7190f4ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:19 UTC) from {username='slave'} at 127.0.0.1:55076
I20260924 03:01:19.269109 20405 tablet_service.cc:1514] Processing DeleteTablet for tablet 108fcbf5c84448b49466100f7190f4ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-24 03:01:19 UTC) from {username='slave'} at 127.0.0.1:47818
I20260924 03:01:19.269300 20509 tablet_replica.cc:330] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97: stopping tablet replica
I20260924 03:01:19.269342 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 20081
I20260924 03:01:19.269423 20509 raft_consensus.cc:2229] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [term 1 LEADER]: Raft consensus shutting down.
I20260924 03:01:19.269518 20509 raft_consensus.cc:2258] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:01:19.270120 20509 ts_tablet_manager.cc:1905] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260924 03:01:19.271647 20509 ts_tablet_manager.cc:1918] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260924 03:01:19.271713 20509 log.cc:1196] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97: Deleting WAL directory at /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790218647334776-392-0/minicluster-data/ts-2/wal/wals/108fcbf5c84448b49466100f7190f4ed
I20260924 03:01:19.271908 20509 ts_tablet_manager.cc:1939] T 108fcbf5c84448b49466100f7190f4ed P ee1ed8abe6b94cda9a149f4dfba28b97: Deleting consensus metadata
I20260924 03:01:19.272261 20007 catalog_manager.cc:4841] TS ee1ed8abe6b94cda9a149f4dfba28b97 (127.0.98.3:44871): tablet 108fcbf5c84448b49466100f7190f4ed (table test-table [id=7c86d64a861e47d4ae774c7da309661c]) successfully deleted
W20260924 03:01:19.277055 20007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.98.1:44131 (error 108)
I20260924 03:01:19.277104 20508 tablet_replica.cc:330] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5: stopping tablet replica
I20260924 03:01:19.277208 20508 raft_consensus.cc:2229] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260924 03:01:19.277223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 20212
I20260924 03:01:19.277316 20508 raft_consensus.cc:2258] T 108fcbf5c84448b49466100f7190f4ed P 0bc32e635bc7434ea4ffa342e70027a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260924 03:01:19.277940 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:19.277489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.1:44131 (local address 127.0.0.1:39448)
0924 03:01:19.277864 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:19.277907 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354}
W20260924 03:01:19.278499 20007 catalog_manager.cc:4584] TS 4922eb076b7d4011a2d4cc13a12e480d (127.0.98.1:44131): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 108fcbf5c84448b49466100f7190f4ed: Network error: Client connection negotiation failed: client connection to 127.0.98.1:44131: connect: Connection refused (error 111)
I20260924 03:01:19.282181 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 20343
I20260924 03:01:19.282248 20482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:19.282150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.2:45679 (local address 127.0.0.1:55078)
0924 03:01:19.282194 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:19.282218 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.2:45679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260924 03:01:19.286079 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 19990
2026-09-24T03:01:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (1022 ms)
[----------] 6 tests from EncryptionPolicyTest (8108 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:19 dist-test-slave-sb6p krb5kdc[20513](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:19 dist-test-slave-sb6p krb5kdc[20513](info): set up 2 sockets
Sep 24 03:01:19 dist-test-slave-sb6p krb5kdc[20513](info): commencing operation
krb5kdc: starting...
W20260924 03:01:20.234174 392 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:20 dist-test-slave-sb6p krb5kdc[20513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:01:20.343219 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43979
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:20.421593 20526 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:20.421790 20526 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:20.421813 20526 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:20.422991 20526 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:20.423032 20526 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:20.423045 20526 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:20.423056 20526 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:20.423067 20526 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:20.424576 20526 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43979
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20526
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:20.424798 20526 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:20.425011 20526 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:20.429636 20534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:20.429666 20532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:20.429685 20531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:20.429800 20526 server_base.cc:1034] running on GCE node
I20260924 03:01:20.430162 20526 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:20.430418 20526 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:20.431565 20526 hybrid_clock.cc:648] HybridClock initialized: now 1790218880431553 us; error 26 us; skew 500 ppm
Sep 24 03:01:20 dist-test-slave-sb6p krb5kdc[20513](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218880, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:20.433895 20526 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:01:20.434309 20526 webserver.cc:460] Webserver started at http://127.0.0.1:42881/ using document root <none> and password file <none>
I20260924 03:01:20.434563 20526 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:20.434617 20526 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:20.434739 20526 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:20.435750 20526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "1e13e9d2021e4dd6ac93ae96fd2b9109"
format_stamp: "Formatted at 2026-09-24 03:01:20 on dist-test-slave-sb6p"
server_key: "28306a51c575a1979d6fffb5f5b09272"
server_key_iv: "b11ccee0be8c8d9095748fe7c4f24733"
server_key_version: "encryptionkey@0"
I20260924 03:01:20.436076 20526 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "1e13e9d2021e4dd6ac93ae96fd2b9109"
format_stamp: "Formatted at 2026-09-24 03:01:20 on dist-test-slave-sb6p"
server_key: "28306a51c575a1979d6fffb5f5b09272"
server_key_iv: "b11ccee0be8c8d9095748fe7c4f24733"
server_key_version: "encryptionkey@0"
I20260924 03:01:20.437305 20526 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:01:20.438112 20541 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:20.438269 20526 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:01:20.438359 20526 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "1e13e9d2021e4dd6ac93ae96fd2b9109"
format_stamp: "Formatted at 2026-09-24 03:01:20 on dist-test-slave-sb6p"
server_key: "28306a51c575a1979d6fffb5f5b09272"
server_key_iv: "b11ccee0be8c8d9095748fe7c4f24733"
server_key_version: "encryptionkey@0"
I20260924 03:01:20.438424 20526 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:20.455605 20526 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:20.457808 20526 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:20.457955 20526 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:20.461747 20526 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43979
I20260924 03:01:20.461783 20593 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43979 every 8 connection(s)
I20260924 03:01:20.462111 20526 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:20.462985 20594 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:20.465235 20594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109: Bootstrap starting.
I20260924 03:01:20.465794 20594 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:20.466055 20594 log.cc:824] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:20.466759 20594 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109: No bootstrap required, opened a new log
I20260924 03:01:20.467605 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 20526
I20260924 03:01:20.467713 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:20.467878 392 external_mini_cluster.cc:1428] Setting key 021a407bef5f8bbdb745d59fdf9ab858
I20260924 03:01:20.468549 20594 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e13e9d2021e4dd6ac93ae96fd2b9109" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43979 } }
I20260924 03:01:20.468719 20594 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:20.468751 20594 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1e13e9d2021e4dd6ac93ae96fd2b9109, State: Initialized, Role: FOLLOWER
I20260924 03:01:20.468871 20594 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e13e9d2021e4dd6ac93ae96fd2b9109" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43979 } }
I20260924 03:01:20.468945 20594 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:20.468977 20594 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:20.469063 20594 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:20.470010 20594 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e13e9d2021e4dd6ac93ae96fd2b9109" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43979 } }
I20260924 03:01:20.470113 20594 leader_election.cc:302] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1e13e9d2021e4dd6ac93ae96fd2b9109; no voters:
I20260924 03:01:20.470255 20594 leader_election.cc:288] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:20.470290 20599 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:20.470445 20599 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [term 1 LEADER]: Becoming Leader. State: Replica: 1e13e9d2021e4dd6ac93ae96fd2b9109, State: Running, Role: LEADER
I20260924 03:01:20.470492 20594 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:20.470624 20599 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e13e9d2021e4dd6ac93ae96fd2b9109" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43979 } }
I20260924 03:01:20.470954 20599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1e13e9d2021e4dd6ac93ae96fd2b9109. Latest consensus state: current_term: 1 leader_uuid: "1e13e9d2021e4dd6ac93ae96fd2b9109" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e13e9d2021e4dd6ac93ae96fd2b9109" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43979 } } }
I20260924 03:01:20.471184 20599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:20.471112 20601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1e13e9d2021e4dd6ac93ae96fd2b9109" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1e13e9d2021e4dd6ac93ae96fd2b9109" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43979 } } }
I20260924 03:01:20.471570 20601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:20.471741 20608 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:20.472319 20608 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:20.473765 20608 catalog_manager.cc:1347] Generated new cluster ID: e9069ccfbfb244d9b9a4abcb0674fd83
I20260924 03:01:20.473814 20608 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:01:20 dist-test-slave-sb6p krb5kdc[20513](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:01:20.479106 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:20.468429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40763 (local address 127.0.0.1:43979)
0924 03:01:20.468661 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:20.468671 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:20.468704 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:01:20.468828 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:20.468835 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:20.468890 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:20.469045 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:20.469656 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:20.471114 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:20.472149 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:20.472731 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:20.472837 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:20.475595 (+ 2758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:20.475600 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:20.475600 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:20.475611 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:20.477447 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:20.477873 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:20.477876 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:20.477876 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:20.477997 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:20.478224 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:20.478224 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:20.478225 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:20.478510 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:20.478675 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:20.478810 (+ 135us) server_negotiation.cc:299] Negotiation successful
0924 03:01:20.478906 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":93,"threads_started":1}
I20260924 03:01:20.480088 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:20.482214 20608 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:20.482923 20608 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:20.487219 20598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:20.481501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34541 (local address 127.0.0.1:43979)
0924 03:01:20.481539 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:20.481541 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:20.481693 (+ 152us) server_negotiation.cc:407] Connection header received
0924 03:01:20.481736 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:20.481736 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:20.481746 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:20.481799 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:20.482371 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:20.482902 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:20.483558 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:20.483779 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:20.483831 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:20.484958 (+ 1127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:20.484960 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:20.484960 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:20.484967 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:20.486182 (+ 1215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:20.486498 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:20.486499 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:20.486500 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:20.486578 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:20.486772 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:20.486773 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:20.486774 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:20.486873 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:20.486959 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:20.487077 (+ 118us) server_negotiation.cc:299] Negotiation successful
0924 03:01:20.487135 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260924 03:01:20.490921 20608 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1e13e9d2021e4dd6ac93ae96fd2b9109: Generated new TSK 0
I20260924 03:01:20.491091 20608 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:20.495889 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 20526
W20260924 03:01:20.499586 20511 connection.cc:535] client connection to 127.0.0.1:43979 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43979): Connection reset by peer (error 104)
2026-09-24T03:01:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:20 dist-test-slave-sb6p krb5kdc[20625](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:20 dist-test-slave-sb6p krb5kdc[20625](info): set up 2 sockets
Sep 24 03:01:20 dist-test-slave-sb6p krb5kdc[20625](info): commencing operation
krb5kdc: starting...
W20260924 03:01:21.446348 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:21 dist-test-slave-sb6p krb5kdc[20625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:01:21.558490 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38627
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33347
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38627
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:21.636389 20638 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:21.636575 20638 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:21.636597 20638 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:21.637702 20638 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:21.637741 20638 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:21.637753 20638 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:21.637766 20638 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:21.637789 20638 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:21.639245 20638 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33347
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38627
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20638
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:21.639463 20638 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:21.639660 20638 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:21.644585 20646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:21.644622 20643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:21.644606 20644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:21.644754 20638 server_base.cc:1034] running on GCE node
I20260924 03:01:21.644977 20638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:21.645221 20638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:21.646368 20638 hybrid_clock.cc:648] HybridClock initialized: now 1790218881646353 us; error 30 us; skew 500 ppm
Sep 24 03:01:21 dist-test-slave-sb6p krb5kdc[20625](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:21.656600 20638 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:01:21.657065 20638 webserver.cc:460] Webserver started at http://127.0.0.1:35031/ using document root <none> and password file <none>
I20260924 03:01:21.657277 20638 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:21.657326 20638 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:21.657450 20638 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:21.658494 20638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "9e4979ffdbf34cef929d2fe92d830d7d"
format_stamp: "Formatted at 2026-09-24 03:01:21 on dist-test-slave-sb6p"
server_key: "476eef002a19701f29ef4cd49499c281"
server_key_iv: "0ce8e62298572296f6bea9bc89c32248"
server_key_version: "encryptionkey@0"
I20260924 03:01:21.658871 20638 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "9e4979ffdbf34cef929d2fe92d830d7d"
format_stamp: "Formatted at 2026-09-24 03:01:21 on dist-test-slave-sb6p"
server_key: "476eef002a19701f29ef4cd49499c281"
server_key_iv: "0ce8e62298572296f6bea9bc89c32248"
server_key_version: "encryptionkey@0"
I20260924 03:01:21.660121 20638 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:01:21.660831 20653 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:21.661000 20638 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:21.661091 20638 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "9e4979ffdbf34cef929d2fe92d830d7d"
format_stamp: "Formatted at 2026-09-24 03:01:21 on dist-test-slave-sb6p"
server_key: "476eef002a19701f29ef4cd49499c281"
server_key_iv: "0ce8e62298572296f6bea9bc89c32248"
server_key_version: "encryptionkey@0"
I20260924 03:01:21.661159 20638 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:21.680511 20638 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:21.682443 20638 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:21.682623 20638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:21.686272 20638 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38627
I20260924 03:01:21.686327 20705 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38627 every 8 connection(s)
I20260924 03:01:21.686668 20638 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:21.687647 20706 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:21.689932 20706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d: Bootstrap starting.
I20260924 03:01:21.690450 20706 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:21.690727 20706 log.cc:824] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:21.691432 20706 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d: No bootstrap required, opened a new log
I20260924 03:01:21.692862 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 20638
I20260924 03:01:21.692970 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:21.692961 20706 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4979ffdbf34cef929d2fe92d830d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38627 } }
I20260924 03:01:21.693127 392 external_mini_cluster.cc:1428] Setting key 6d44c52a00335a3503c566febeb3e8ab
I20260924 03:01:21.693135 20706 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:21.693176 20706 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e4979ffdbf34cef929d2fe92d830d7d, State: Initialized, Role: FOLLOWER
I20260924 03:01:21.693318 20706 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4979ffdbf34cef929d2fe92d830d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38627 } }
I20260924 03:01:21.693406 20706 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:21.693437 20706 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:21.693501 20706 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:21.694096 20706 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4979ffdbf34cef929d2fe92d830d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38627 } }
I20260924 03:01:21.694205 20706 leader_election.cc:302] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e4979ffdbf34cef929d2fe92d830d7d; no voters:
I20260924 03:01:21.694406 20706 leader_election.cc:288] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:21.694442 20711 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:21.694686 20711 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [term 1 LEADER]: Becoming Leader. State: Replica: 9e4979ffdbf34cef929d2fe92d830d7d, State: Running, Role: LEADER
I20260924 03:01:21.694697 20706 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:21.694849 20711 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4979ffdbf34cef929d2fe92d830d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38627 } }
I20260924 03:01:21.695214 20712 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e4979ffdbf34cef929d2fe92d830d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4979ffdbf34cef929d2fe92d830d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38627 } } }
I20260924 03:01:21.695286 20712 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:21.695488 20713 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e4979ffdbf34cef929d2fe92d830d7d. Latest consensus state: current_term: 1 leader_uuid: "9e4979ffdbf34cef929d2fe92d830d7d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e4979ffdbf34cef929d2fe92d830d7d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38627 } } }
I20260924 03:01:21.695760 20713 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:21.695885 20718 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:21.696386 20718 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:21.697844 20718 catalog_manager.cc:1347] Generated new cluster ID: ca450974f2e54734b67d1358163e9174
I20260924 03:01:21.697909 20718 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:21.699059 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:21.693569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44629 (local address 127.0.0.1:38627)
0924 03:01:21.693754 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:21.693762 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:21.693787 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:01:21.694120 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:21.694125 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:21.694171 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:21.694302 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:21.694619 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:21.696016 (+ 1397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:21.697075 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:21.697601 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:21.697711 (+ 110us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:21.698372 (+ 661us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:21.698376 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:21.698376 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:21.698635 (+ 259us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:21.698667 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:21.698679 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:21.698749 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:21.698836 (+ 87us) server_negotiation.cc:299] Negotiation successful
0924 03:01:21.698862 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":91,"threads_started":1}
I20260924 03:01:21.699877 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:21.703703 20710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:21.701116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40561 (local address 127.0.0.1:38627)
0924 03:01:21.701170 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:21.701172 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:21.701206 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:01:21.701270 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:21.701271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:21.701280 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:21.701320 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:21.701431 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:21.701900 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:21.702628 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:21.702848 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:21.702901 (+ 53us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:21.703482 (+ 581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:21.703483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:21.703483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:21.703488 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:21.703502 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:21.703506 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:21.703553 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:21.703618 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 03:01:21.703628 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260924 03:01:21.714205 20718 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:21.714747 20718 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:21.717211 20718 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e4979ffdbf34cef929d2fe92d830d7d: Generated new TSK 0
I20260924 03:01:21.717358 20718 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:21.723827 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 20638
W20260924 03:01:21.727474 20623 connection.cc:535] client connection to 127.0.0.1:38627 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38627): Connection reset by peer (error 104)
2026-09-24T03:01:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1227 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:21 dist-test-slave-sb6p krb5kdc[20737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:21 dist-test-slave-sb6p krb5kdc[20737](info): set up 2 sockets
Sep 24 03:01:21 dist-test-slave-sb6p krb5kdc[20737](info): commencing operation
krb5kdc: starting...
W20260924 03:01:22.666208 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:22 dist-test-slave-sb6p krb5kdc[20737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218882, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:01:22.776683 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41641
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37795
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41641
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:22.855537 20750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:22.855738 20750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:22.855768 20750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:22.856902 20750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:22.856941 20750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:22.856958 20750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:22.856981 20750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:22.856999 20750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:22.858422 20750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37795
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41641
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:22.858677 20750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:22.858875 20750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:22.863526 20758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:22.863528 20756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:22.863560 20750 server_base.cc:1034] running on GCE node
W20260924 03:01:22.863543 20755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:22.863974 20750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:22.864241 20750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:22.865391 20750 hybrid_clock.cc:648] HybridClock initialized: now 1790218882865377 us; error 25 us; skew 500 ppm
Sep 24 03:01:22 dist-test-slave-sb6p krb5kdc[20737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:22.867762 20750 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:01:22.868151 20750 webserver.cc:460] Webserver started at http://127.0.0.1:36021/ using document root <none> and password file <none>
I20260924 03:01:22.868340 20750 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:22.868396 20750 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:22.868522 20750 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:22.869527 20750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "af9cfc4582944adebea002bf0c63d5fe"
format_stamp: "Formatted at 2026-09-24 03:01:22 on dist-test-slave-sb6p"
server_key: "596584dff76f4343451ace47b0d0ac30"
server_key_iv: "9cc9767550ec6912a9872034fcae3bf7"
server_key_version: "encryptionkey@0"
I20260924 03:01:22.869843 20750 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "af9cfc4582944adebea002bf0c63d5fe"
format_stamp: "Formatted at 2026-09-24 03:01:22 on dist-test-slave-sb6p"
server_key: "596584dff76f4343451ace47b0d0ac30"
server_key_iv: "9cc9767550ec6912a9872034fcae3bf7"
server_key_version: "encryptionkey@0"
I20260924 03:01:22.871156 20750 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:01:22.871879 20765 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:22.872018 20750 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:22.872102 20750 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "af9cfc4582944adebea002bf0c63d5fe"
format_stamp: "Formatted at 2026-09-24 03:01:22 on dist-test-slave-sb6p"
server_key: "596584dff76f4343451ace47b0d0ac30"
server_key_iv: "9cc9767550ec6912a9872034fcae3bf7"
server_key_version: "encryptionkey@0"
I20260924 03:01:22.872161 20750 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:22.875344 20750 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:22.877056 20750 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:22.877177 20750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:22.880703 20750 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41641
I20260924 03:01:22.880733 20817 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41641 every 8 connection(s)
I20260924 03:01:22.881014 20750 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:22.881881 20818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:22.884042 20818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe: Bootstrap starting.
I20260924 03:01:22.884560 20818 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:22.884821 20818 log.cc:824] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:22.885393 20818 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe: No bootstrap required, opened a new log
I20260924 03:01:22.886974 20818 raft_consensus.cc:348] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9cfc4582944adebea002bf0c63d5fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41641 } }
I20260924 03:01:22.887099 20818 raft_consensus.cc:374] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:22.887120 20818 raft_consensus.cc:729] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9cfc4582944adebea002bf0c63d5fe, State: Initialized, Role: FOLLOWER
I20260924 03:01:22.887243 20818 consensus_queue.cc:260] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9cfc4582944adebea002bf0c63d5fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41641 } }
I20260924 03:01:22.887300 20818 raft_consensus.cc:388] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:22.887321 20818 raft_consensus.cc:482] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:22.887364 20818 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:22.887840 20818 raft_consensus.cc:504] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9cfc4582944adebea002bf0c63d5fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41641 } }
I20260924 03:01:22.887926 20818 leader_election.cc:302] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: af9cfc4582944adebea002bf0c63d5fe; no voters:
I20260924 03:01:22.888065 20818 leader_election.cc:288] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:22.888108 20821 raft_consensus.cc:2781] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:22.888289 20818 sys_catalog.cc:564] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:22.888428 20821 raft_consensus.cc:686] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [term 1 LEADER]: Becoming Leader. State: Replica: af9cfc4582944adebea002bf0c63d5fe, State: Running, Role: LEADER
I20260924 03:01:22.888554 20821 consensus_queue.cc:237] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9cfc4582944adebea002bf0c63d5fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41641 } }
I20260924 03:01:22.888892 20823 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader af9cfc4582944adebea002bf0c63d5fe. Latest consensus state: current_term: 1 leader_uuid: "af9cfc4582944adebea002bf0c63d5fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9cfc4582944adebea002bf0c63d5fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41641 } } }
I20260924 03:01:22.888885 20822 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "af9cfc4582944adebea002bf0c63d5fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "af9cfc4582944adebea002bf0c63d5fe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41641 } } }
I20260924 03:01:22.888972 20823 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:22.888978 20822 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:22.889197 20830 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:22.889706 20830 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:22.890712 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 20750
I20260924 03:01:22.890825 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:22.890972 392 external_mini_cluster.cc:1428] Setting key 734faef5dd4569696f30e46d9afa861a
I20260924 03:01:22.891654 20830 catalog_manager.cc:1347] Generated new cluster ID: 5ae9a1099f6f4707b76ee5e781f1d311
I20260924 03:01:22.891702 20830 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:22.893123 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:22.891409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59301 (local address 127.0.0.1:41641)
0924 03:01:22.891729 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:22.891738 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:22.891768 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:01:22.891878 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:22.891886 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:22.891915 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:22.892051 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:22.892624 (+ 573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:22.892627 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:22.892627 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:22.892631 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:22.892671 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:22.892686 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:22.892754 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:22.892809 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 03:01:22.892938 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":280,"thread_start_us":71,"threads_started":1}
I20260924 03:01:22.893924 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:22.896246 20841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:22.895361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36265 (local address 127.0.0.1:41641)
0924 03:01:22.895394 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:22.895395 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:22.895403 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:01:22.895460 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:22.895461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:22.895465 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:22.895534 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:22.896046 (+ 512us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:22.896047 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:22.896048 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:22.896052 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:22.896066 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:22.896070 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:22.896130 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:22.896167 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 03:01:22.896179 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:01:22.904030 20830 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:22.904191 20830 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:22.916476 20830 catalog_manager.cc:5868] T 00000000000000000000000000000000 P af9cfc4582944adebea002bf0c63d5fe: Generated new TSK 0
I20260924 03:01:22.916689 20830 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:22.920661 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 20750
2026-09-24T03:01:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1196 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:22 dist-test-slave-sb6p krb5kdc[20849](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:22 dist-test-slave-sb6p krb5kdc[20849](info): set up 2 sockets
Sep 24 03:01:22 dist-test-slave-sb6p krb5kdc[20849](info): commencing operation
krb5kdc: starting...
W20260924 03:01:23.866416 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:23 dist-test-slave-sb6p krb5kdc[20849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:01:23.977744 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45583
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36255
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45583
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:24.055383 20862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:24.055603 20862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:24.055634 20862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:24.056800 20862 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:24.056845 20862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:24.056864 20862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:24.056880 20862 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:24.056895 20862 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:24.058320 20862 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36255
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45583
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45583
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:24.058606 20862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:24.058810 20862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:24.063370 20870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:24.063371 20867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:24.063401 20862 server_base.cc:1034] running on GCE node
W20260924 03:01:24.063365 20868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:24.063841 20862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:24.064105 20862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:24.065265 20862 hybrid_clock.cc:648] HybridClock initialized: now 1790218884065251 us; error 28 us; skew 500 ppm
Sep 24 03:01:24 dist-test-slave-sb6p krb5kdc[20849](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:24.067749 20862 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:01:24.068161 20862 webserver.cc:460] Webserver started at http://127.0.0.1:36137/ using document root <none> and password file <none>
I20260924 03:01:24.068360 20862 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:24.068418 20862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:24.068544 20862 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:24.069533 20862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "4e17502d400946e693c442220f4e65b8"
format_stamp: "Formatted at 2026-09-24 03:01:24 on dist-test-slave-sb6p"
server_key: "dcacc8c7336abd3fdc2791d665da3b85"
server_key_iv: "f85aa6a52f2dfd90d59faa09aa7f6008"
server_key_version: "encryptionkey@0"
I20260924 03:01:24.069846 20862 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "4e17502d400946e693c442220f4e65b8"
format_stamp: "Formatted at 2026-09-24 03:01:24 on dist-test-slave-sb6p"
server_key: "dcacc8c7336abd3fdc2791d665da3b85"
server_key_iv: "f85aa6a52f2dfd90d59faa09aa7f6008"
server_key_version: "encryptionkey@0"
I20260924 03:01:24.071173 20862 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:01:24.072006 20877 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:24.072201 20862 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:01:24.072284 20862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "4e17502d400946e693c442220f4e65b8"
format_stamp: "Formatted at 2026-09-24 03:01:24 on dist-test-slave-sb6p"
server_key: "dcacc8c7336abd3fdc2791d665da3b85"
server_key_iv: "f85aa6a52f2dfd90d59faa09aa7f6008"
server_key_version: "encryptionkey@0"
I20260924 03:01:24.072341 20862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:24.087925 20862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:24.089972 20862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:24.090109 20862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:24.094084 20862 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45583
I20260924 03:01:24.094115 20929 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45583 every 8 connection(s)
I20260924 03:01:24.094429 20862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:24.095281 20930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:24.097482 20930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8: Bootstrap starting.
I20260924 03:01:24.098059 20930 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:24.098312 20930 log.cc:824] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:24.099115 20930 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8: No bootstrap required, opened a new log
I20260924 03:01:24.100708 20930 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e17502d400946e693c442220f4e65b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45583 } }
I20260924 03:01:24.100842 20930 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:24.100863 20930 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e17502d400946e693c442220f4e65b8, State: Initialized, Role: FOLLOWER
I20260924 03:01:24.101006 20930 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e17502d400946e693c442220f4e65b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45583 } }
I20260924 03:01:24.101069 20930 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:24.101089 20930 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:24.101118 20930 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:24.101625 20930 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e17502d400946e693c442220f4e65b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45583 } }
I20260924 03:01:24.101712 20930 leader_election.cc:302] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e17502d400946e693c442220f4e65b8; no voters:
I20260924 03:01:24.101859 20930 leader_election.cc:288] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:24.101920 20933 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:24.102068 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 20862
I20260924 03:01:24.102113 20930 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:24.102151 20933 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [term 1 LEADER]: Becoming Leader. State: Replica: 4e17502d400946e693c442220f4e65b8, State: Running, Role: LEADER
I20260924 03:01:24.102181 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:24.102275 20933 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e17502d400946e693c442220f4e65b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45583 } }
I20260924 03:01:24.102345 392 external_mini_cluster.cc:1428] Setting key f686e2ed19409715f60dbbfc4ff011af
I20260924 03:01:24.102861 20934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4e17502d400946e693c442220f4e65b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e17502d400946e693c442220f4e65b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45583 } } }
I20260924 03:01:24.102942 20934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:24.103106 20935 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4e17502d400946e693c442220f4e65b8. Latest consensus state: current_term: 1 leader_uuid: "4e17502d400946e693c442220f4e65b8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e17502d400946e693c442220f4e65b8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45583 } } }
I20260924 03:01:24.103214 20935 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:24.103317 20944 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:24.103892 20944 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:24.105381 20944 catalog_manager.cc:1347] Generated new cluster ID: 68745660464040f78d8fd2f4d046f4fe
I20260924 03:01:24.105432 20944 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:01:24 dist-test-slave-sb6p krb5kdc[20849](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218883, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:01:24.112434 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:24.102861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36255 (local address 127.0.0.1:45583)
0924 03:01:24.103094 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:24.103104 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:24.103524 (+ 420us) server_negotiation.cc:407] Connection header received
0924 03:01:24.103655 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:24.103663 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:24.103729 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:24.103885 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:24.104577 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:24.105626 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:24.106390 (+ 764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:24.106884 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:24.106989 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:24.109115 (+ 2126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:24.109120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:24.109120 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:24.109133 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:24.111088 (+ 1955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:24.111392 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:24.111395 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:24.111395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:24.111492 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:24.111662 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:24.111663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:24.111663 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:24.111888 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:24.112042 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:24.112134 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 03:01:24.112264 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":67,"threads_started":1}
I20260924 03:01:24.113230 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:24.118513 20944 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:24.119275 20944 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:24.125020 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:24.114455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50741 (local address 127.0.0.1:45583)
0924 03:01:24.114488 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:24.114490 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:24.118420 (+ 3930us) server_negotiation.cc:407] Connection header received
0924 03:01:24.118501 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:24.118502 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:24.118512 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:24.118583 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:24.119195 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:24.119951 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:24.120893 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:24.121201 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:24.121268 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:24.122549 (+ 1281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:24.122551 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:24.122551 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:24.122558 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:24.124010 (+ 1452us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:24.124339 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:24.124340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:24.124341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:24.124396 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:24.124550 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:24.124550 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:24.124550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:24.124670 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:24.124771 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:24.124843 (+ 72us) server_negotiation.cc:299] Negotiation successful
0924 03:01:24.124930 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:01:24.131812 20944 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4e17502d400946e693c442220f4e65b8: Generated new TSK 0
I20260924 03:01:24.132004 20944 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:24.135195 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 20862
2026-09-24T03:01:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:24 dist-test-slave-sb6p krb5kdc[20961](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:24 dist-test-slave-sb6p krb5kdc[20961](info): set up 2 sockets
Sep 24 03:01:24 dist-test-slave-sb6p krb5kdc[20961](info): commencing operation
krb5kdc: starting...
W20260924 03:01:25.074234 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.002s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:25 dist-test-slave-sb6p krb5kdc[20961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218885, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:01:25.184643 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33221
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33895
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33221
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:25.263324 20974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:25.263540 20974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:25.263571 20974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:25.264745 20974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:25.264787 20974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:25.264806 20974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:25.264822 20974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:25.264879 20974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:25.266341 20974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33895
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33221
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20974
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:25.266601 20974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:25.266798 20974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:25.271382 20982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:25.271400 20980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:25.271415 20974 server_base.cc:1034] running on GCE node
W20260924 03:01:25.271374 20979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:25.271816 20974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:25.272078 20974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:25.273234 20974 hybrid_clock.cc:648] HybridClock initialized: now 1790218885273224 us; error 29 us; skew 500 ppm
Sep 24 03:01:25 dist-test-slave-sb6p krb5kdc[20961](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:25.275705 20974 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:01:25.276101 20974 webserver.cc:460] Webserver started at http://127.0.0.1:37125/ using document root <none> and password file <none>
I20260924 03:01:25.276288 20974 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:25.276333 20974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:25.276458 20974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:25.277458 20974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "777953144a524e98ba36be2f74c851e3"
format_stamp: "Formatted at 2026-09-24 03:01:25 on dist-test-slave-sb6p"
server_key: "e6f682901e8a7f3081c7b75326dfe308"
server_key_iv: "431e842b21a143e89592c1c07ea79ca2"
server_key_version: "encryptionkey@0"
I20260924 03:01:25.277765 20974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "777953144a524e98ba36be2f74c851e3"
format_stamp: "Formatted at 2026-09-24 03:01:25 on dist-test-slave-sb6p"
server_key: "e6f682901e8a7f3081c7b75326dfe308"
server_key_iv: "431e842b21a143e89592c1c07ea79ca2"
server_key_version: "encryptionkey@0"
I20260924 03:01:25.279091 20974 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:01:25.279837 20989 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:25.279991 20974 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:01:25.280076 20974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "777953144a524e98ba36be2f74c851e3"
format_stamp: "Formatted at 2026-09-24 03:01:25 on dist-test-slave-sb6p"
server_key: "e6f682901e8a7f3081c7b75326dfe308"
server_key_iv: "431e842b21a143e89592c1c07ea79ca2"
server_key_version: "encryptionkey@0"
I20260924 03:01:25.280133 20974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:25.293077 20974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:25.294906 20974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:25.295044 20974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:25.298553 20974 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33221
I20260924 03:01:25.298616 21041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33221 every 8 connection(s)
I20260924 03:01:25.298941 20974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:25.299803 21042 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:25.302031 21042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3: Bootstrap starting.
I20260924 03:01:25.302596 21042 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:25.302839 21042 log.cc:824] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:25.303556 21042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3: No bootstrap required, opened a new log
I20260924 03:01:25.305088 21042 raft_consensus.cc:348] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777953144a524e98ba36be2f74c851e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33221 } }
I20260924 03:01:25.305222 21042 raft_consensus.cc:374] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:25.305243 21042 raft_consensus.cc:729] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 777953144a524e98ba36be2f74c851e3, State: Initialized, Role: FOLLOWER
I20260924 03:01:25.305372 21042 consensus_queue.cc:260] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777953144a524e98ba36be2f74c851e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33221 } }
I20260924 03:01:25.305440 21042 raft_consensus.cc:388] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:25.305462 21042 raft_consensus.cc:482] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:25.305507 21042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:25.305986 21042 raft_consensus.cc:504] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777953144a524e98ba36be2f74c851e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33221 } }
I20260924 03:01:25.306071 21042 leader_election.cc:302] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 777953144a524e98ba36be2f74c851e3; no voters:
I20260924 03:01:25.306216 21042 leader_election.cc:288] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:25.306272 21045 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:25.306418 21045 raft_consensus.cc:686] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [term 1 LEADER]: Becoming Leader. State: Replica: 777953144a524e98ba36be2f74c851e3, State: Running, Role: LEADER
I20260924 03:01:25.306444 21042 sys_catalog.cc:564] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:25.306509 21045 consensus_queue.cc:237] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777953144a524e98ba36be2f74c851e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33221 } }
I20260924 03:01:25.306864 21046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "777953144a524e98ba36be2f74c851e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777953144a524e98ba36be2f74c851e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33221 } } }
I20260924 03:01:25.306864 21047 sys_catalog.cc:455] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 777953144a524e98ba36be2f74c851e3. Latest consensus state: current_term: 1 leader_uuid: "777953144a524e98ba36be2f74c851e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "777953144a524e98ba36be2f74c851e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33221 } } }
I20260924 03:01:25.307020 21046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:25.307029 21047 sys_catalog.cc:458] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:25.307427 21049 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:25.307874 21049 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:25.308820 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 20974
I20260924 03:01:25.308930 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:25.309096 392 external_mini_cluster.cc:1428] Setting key ccdca8ba34a0551aabed9d790cf5c922
I20260924 03:01:25.309484 21049 catalog_manager.cc:1347] Generated new cluster ID: 1dd4ce1fe4d6452884816029785e64c2
I20260924 03:01:25.309532 21049 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:25.313150 21049 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:25.313681 21049 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:25.318499 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:25.309682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33907 (local address 127.0.0.1:33221)
0924 03:01:25.312799 (+ 3117us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:25.312808 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:25.312843 (+ 35us) server_negotiation.cc:407] Connection header received
0924 03:01:25.312959 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:25.312964 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:25.313015 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:25.313808 (+ 793us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:25.313893 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:25.315398 (+ 1505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:25.316507 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:25.317117 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:25.317225 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:25.317847 (+ 622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:25.317851 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:25.317852 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:25.318052 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:25.318090 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:25.318104 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:25.318170 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:25.318223 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 03:01:25.318282 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3073,"thread_start_us":77,"threads_started":1}
I20260924 03:01:25.319358 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:25.324029 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:25.320682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43863 (local address 127.0.0.1:33221)
0924 03:01:25.320723 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:25.320725 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:25.320879 (+ 154us) server_negotiation.cc:407] Connection header received
0924 03:01:25.320922 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:25.320923 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:25.320933 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:25.320987 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:25.321082 (+ 95us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:25.321823 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:25.322832 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:25.323132 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:25.323195 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:25.323800 (+ 605us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:25.323802 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:25.323802 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:25.323806 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:25.323821 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:25.323825 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:25.323873 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:25.323931 (+ 58us) server_negotiation.cc:299] Negotiation successful
0924 03:01:25.323955 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:01:25.324780 21049 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 777953144a524e98ba36be2f74c851e3: Generated new TSK 0
I20260924 03:01:25.324930 21049 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:25.328724 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 20974
2026-09-24T03:01:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1193 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:25 dist-test-slave-sb6p krb5kdc[21073](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:25 dist-test-slave-sb6p krb5kdc[21073](info): set up 2 sockets
Sep 24 03:01:25 dist-test-slave-sb6p krb5kdc[21073](info): commencing operation
krb5kdc: starting...
W20260924 03:01:26.266360 392 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:26 dist-test-slave-sb6p krb5kdc[21073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218886, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:01:26.378036 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33335
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40285
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33335
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:26.456403 21086 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:26.456668 21086 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:26.456712 21086 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:26.457913 21086 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:26.457958 21086 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:26.457978 21086 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:26.457994 21086 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:26.458009 21086 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:26.459578 21086 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40285
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33335
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21086
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:26.459824 21086 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:26.460021 21086 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:26.464707 21094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:26.464715 21091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:26.464847 21092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:26.464926 21086 server_base.cc:1034] running on GCE node
I20260924 03:01:26.465180 21086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:26.465436 21086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:26.466598 21086 hybrid_clock.cc:648] HybridClock initialized: now 1790218886466581 us; error 32 us; skew 500 ppm
Sep 24 03:01:26 dist-test-slave-sb6p krb5kdc[21073](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218886, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:26.469050 21086 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:01:26.469465 21086 webserver.cc:460] Webserver started at http://127.0.0.1:42627/ using document root <none> and password file <none>
I20260924 03:01:26.469635 21086 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:26.469673 21086 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:26.469761 21086 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:26.470793 21086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "1dbb6299e1cb43fb89bcb509f5802a97"
format_stamp: "Formatted at 2026-09-24 03:01:26 on dist-test-slave-sb6p"
server_key: "ee9bf1c66986f8ffb64829bc727144c4"
server_key_iv: "a14dcfb9c5c286915d1417ff82711a27"
server_key_version: "encryptionkey@0"
I20260924 03:01:26.471083 21086 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "1dbb6299e1cb43fb89bcb509f5802a97"
format_stamp: "Formatted at 2026-09-24 03:01:26 on dist-test-slave-sb6p"
server_key: "ee9bf1c66986f8ffb64829bc727144c4"
server_key_iv: "a14dcfb9c5c286915d1417ff82711a27"
server_key_version: "encryptionkey@0"
I20260924 03:01:26.472203 21086 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:01:26.472843 21101 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:26.473002 21086 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:26.473073 21086 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "1dbb6299e1cb43fb89bcb509f5802a97"
format_stamp: "Formatted at 2026-09-24 03:01:26 on dist-test-slave-sb6p"
server_key: "ee9bf1c66986f8ffb64829bc727144c4"
server_key_iv: "a14dcfb9c5c286915d1417ff82711a27"
server_key_version: "encryptionkey@0"
I20260924 03:01:26.473122 21086 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:26.476311 21086 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:26.477978 21086 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:26.478082 21086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:26.481642 21086 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33335
I20260924 03:01:26.481673 21153 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33335 every 8 connection(s)
I20260924 03:01:26.481976 21086 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:26.482164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 21086
I20260924 03:01:26.482268 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:26.482419 392 external_mini_cluster.cc:1428] Setting key c4b1dbec43acd2d59c620396585b6eee
I20260924 03:01:26.483089 21154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:26.484846 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:26.482975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46421 (local address 127.0.0.1:33335)
0924 03:01:26.483178 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:26.483188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:26.483219 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:01:26.483715 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:26.483724 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:26.483758 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:26.483903 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:26.484457 (+ 554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:26.484461 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:26.484461 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:26.484465 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:26.484507 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:26.484521 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:26.484591 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:26.484624 (+ 33us) server_negotiation.cc:299] Negotiation successful
0924 03:01:26.484663 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":89,"threads_started":1}
I20260924 03:01:26.485769 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:26.485999 21154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97: Bootstrap starting.
I20260924 03:01:26.486575 21154 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:26.486852 21154 log.cc:824] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:26.487519 21154 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97: No bootstrap required, opened a new log
I20260924 03:01:26.488063 21157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:26.487045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51037 (local address 127.0.0.1:33335)
0924 03:01:26.487081 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:26.487082 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:26.487244 (+ 162us) server_negotiation.cc:407] Connection header received
0924 03:01:26.487289 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:26.487290 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:26.487295 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:26.487367 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:26.487871 (+ 504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:26.487872 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:26.487872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:26.487876 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:26.487888 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:26.487891 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:26.487940 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:26.488011 (+ 71us) server_negotiation.cc:299] Negotiation successful
0924 03:01:26.488021 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:01:26.489485 21154 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbb6299e1cb43fb89bcb509f5802a97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } }
I20260924 03:01:26.489614 21154 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:26.489635 21154 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dbb6299e1cb43fb89bcb509f5802a97, State: Initialized, Role: FOLLOWER
I20260924 03:01:26.489733 21154 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbb6299e1cb43fb89bcb509f5802a97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } }
I20260924 03:01:26.489813 21154 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:26.489846 21154 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:26.489892 21154 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:26.490373 21154 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbb6299e1cb43fb89bcb509f5802a97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } }
I20260924 03:01:26.490470 21154 leader_election.cc:302] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1dbb6299e1cb43fb89bcb509f5802a97; no voters:
I20260924 03:01:26.490648 21154 leader_election.cc:288] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:26.490702 21164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:26.490857 21164 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [term 1 LEADER]: Becoming Leader. State: Replica: 1dbb6299e1cb43fb89bcb509f5802a97, State: Running, Role: LEADER
I20260924 03:01:26.490896 21154 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:26.490952 21164 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbb6299e1cb43fb89bcb509f5802a97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } }
I20260924 03:01:26.491314 21165 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1dbb6299e1cb43fb89bcb509f5802a97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbb6299e1cb43fb89bcb509f5802a97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } } }
I20260924 03:01:26.491405 21165 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:26.491499 21166 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1dbb6299e1cb43fb89bcb509f5802a97. Latest consensus state: current_term: 1 leader_uuid: "1dbb6299e1cb43fb89bcb509f5802a97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1dbb6299e1cb43fb89bcb509f5802a97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33335 } } }
I20260924 03:01:26.491742 21166 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:26.491910 21173 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:26.492364 21173 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:26.493798 21173 catalog_manager.cc:1347] Generated new cluster ID: 997064e2daf740558b48513621db391a
I20260924 03:01:26.493836 21173 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:26.509965 21173 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:26.510134 21173 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:26.520546 21173 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1dbb6299e1cb43fb89bcb509f5802a97: Generated new TSK 0
I20260924 03:01:26.520717 21173 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:26.528484 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 21086
2026-09-24T03:01:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1199 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:26 dist-test-slave-sb6p krb5kdc[21185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:26 dist-test-slave-sb6p krb5kdc[21185](info): set up 2 sockets
Sep 24 03:01:26 dist-test-slave-sb6p krb5kdc[21185](info): commencing operation
krb5kdc: starting...
W20260924 03:01:27.470314 392 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:27 dist-test-slave-sb6p krb5kdc[21185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:01:27.584671 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34259
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:27.662694 21198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:27.662889 21198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:27.662911 21198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:27.664161 21198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:27.664207 21198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:27.664222 21198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:27.664232 21198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:27.664243 21198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:27.665699 21198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34259
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40607
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:27.665916 21198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:27.666129 21198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260924 03:01:27.670851 21198 server_base.cc:1034] running on GCE node
W20260924 03:01:27.670818 21206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:27.670825 21204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:27.670818 21203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:27.671268 21198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:27.671573 21198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:27.672729 21198 hybrid_clock.cc:648] HybridClock initialized: now 1790218887672713 us; error 29 us; skew 500 ppm
Sep 24 03:01:27 dist-test-slave-sb6p krb5kdc[21185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:27.680557 21198 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:01:27.681017 21198 webserver.cc:460] Webserver started at http://127.0.0.1:37889/ using document root <none> and password file <none>
I20260924 03:01:27.681241 21198 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:27.681284 21198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:27.681404 21198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:27.682466 21198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "2e8c78e56bd14ea3bc3a3c7e1c156af3"
format_stamp: "Formatted at 2026-09-24 03:01:27 on dist-test-slave-sb6p"
server_key: "25afdeccad226fc3c85dd56e1cfa3201"
server_key_iv: "137f62978077c986c38f4e47f3b663b0"
server_key_version: "encryptionkey@0"
I20260924 03:01:27.682845 21198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "2e8c78e56bd14ea3bc3a3c7e1c156af3"
format_stamp: "Formatted at 2026-09-24 03:01:27 on dist-test-slave-sb6p"
server_key: "25afdeccad226fc3c85dd56e1cfa3201"
server_key_iv: "137f62978077c986c38f4e47f3b663b0"
server_key_version: "encryptionkey@0"
I20260924 03:01:27.684104 21198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260924 03:01:27.684803 21213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:27.684969 21198 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:27.685045 21198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "2e8c78e56bd14ea3bc3a3c7e1c156af3"
format_stamp: "Formatted at 2026-09-24 03:01:27 on dist-test-slave-sb6p"
server_key: "25afdeccad226fc3c85dd56e1cfa3201"
server_key_iv: "137f62978077c986c38f4e47f3b663b0"
server_key_version: "encryptionkey@0"
I20260924 03:01:27.685113 21198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:27.699411 21198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:27.701546 21198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:27.701664 21198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:27.705722 21198 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40607
I20260924 03:01:27.705757 21265 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40607 every 8 connection(s)
I20260924 03:01:27.706080 21198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:27.706895 21266 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:27.709164 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 21198
I20260924 03:01:27.709257 21266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3: Bootstrap starting.
I20260924 03:01:27.709282 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:27.709450 392 external_mini_cluster.cc:1428] Setting key 0f85f4e6870845e9e277ff4436d0182b
I20260924 03:01:27.709949 21266 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:27.710232 21266 log.cc:824] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:27.710988 21266 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3: No bootstrap required, opened a new log
I20260924 03:01:27.713091 21266 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e8c78e56bd14ea3bc3a3c7e1c156af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } }
I20260924 03:01:27.713258 21266 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:27.713287 21266 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e8c78e56bd14ea3bc3a3c7e1c156af3, State: Initialized, Role: FOLLOWER
I20260924 03:01:27.713409 21266 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e8c78e56bd14ea3bc3a3c7e1c156af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } }
I20260924 03:01:27.713474 21266 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:27.713510 21266 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:27.713598 21266 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:27.714116 21266 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e8c78e56bd14ea3bc3a3c7e1c156af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } }
I20260924 03:01:27.714221 21266 leader_election.cc:302] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e8c78e56bd14ea3bc3a3c7e1c156af3; no voters:
I20260924 03:01:27.714371 21266 leader_election.cc:288] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:27.714402 21271 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:27.714560 21271 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [term 1 LEADER]: Becoming Leader. State: Replica: 2e8c78e56bd14ea3bc3a3c7e1c156af3, State: Running, Role: LEADER
I20260924 03:01:27.714651 21266 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:27.714699 21271 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e8c78e56bd14ea3bc3a3c7e1c156af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } }
I20260924 03:01:27.715118 21272 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e8c78e56bd14ea3bc3a3c7e1c156af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e8c78e56bd14ea3bc3a3c7e1c156af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } } }
I20260924 03:01:27.715194 21272 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:27.715433 21278 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:27.715786 21273 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e8c78e56bd14ea3bc3a3c7e1c156af3. Latest consensus state: current_term: 1 leader_uuid: "2e8c78e56bd14ea3bc3a3c7e1c156af3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e8c78e56bd14ea3bc3a3c7e1c156af3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40607 } } }
I20260924 03:01:27.715960 21273 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3 [sys.catalog]: This master's current role is: LEADER
Sep 24 03:01:27 dist-test-slave-sb6p krb5kdc[21185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218887, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:01:27.716620 21278 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:27.718889 21278 catalog_manager.cc:1347] Generated new cluster ID: f0ab073a40cf414ab233de0d0766d0f4
I20260924 03:01:27.718935 21278 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:27.720523 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:27.709953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:42213 (local address 127.0.0.1:40607)
0924 03:01:27.710126 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:27.710134 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:27.710159 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:01:27.710735 (+ 576us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:27.710743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:27.710806 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:27.710920 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:27.711678 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:27.712580 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:27.713617 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:27.714054 (+ 437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:27.714135 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:27.716861 (+ 2726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:27.716865 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:27.716865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:27.716877 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:27.718825 (+ 1948us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:27.719244 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:27.719247 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:27.719247 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:27.719355 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:27.719648 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:27.719649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:27.719649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:27.719959 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:27.720120 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:27.720211 (+ 91us) server_negotiation.cc:299] Negotiation successful
0924 03:01:27.720310 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":65,"threads_started":1}
I20260924 03:01:27.723686 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:27.725829 21278 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:27.726464 21278 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:27.731801 21278 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e8c78e56bd14ea3bc3a3c7e1c156af3: Generated new TSK 0
I20260924 03:01:27.731973 21278 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:27.732287 21269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:27.725099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:35133 (local address 127.0.0.1:40607)
0924 03:01:27.725139 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:27.725141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:27.725274 (+ 133us) server_negotiation.cc:407] Connection header received
0924 03:01:27.725318 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:27.725319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:27.725329 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:27.725387 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:27.725995 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:27.726719 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:27.727723 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:27.728026 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:27.728087 (+ 61us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:27.729471 (+ 1384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:27.729473 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:27.729473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:27.729480 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:27.730975 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:27.731372 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:27.731372 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:27.731373 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:27.731434 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:27.731674 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:27.731675 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:27.731675 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:27.731816 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:27.731916 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:27.732128 (+ 212us) server_negotiation.cc:299] Negotiation successful
0924 03:01:27.732195 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:01:27.734453 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 21198
W20260924 03:01:27.738350 21183 connection.cc:535] client connection to 127.0.0.1:40607 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40607): Connection reset by peer (error 104)
2026-09-24T03:01:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:27 dist-test-slave-sb6p krb5kdc[21297](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:27 dist-test-slave-sb6p krb5kdc[21297](info): set up 2 sockets
Sep 24 03:01:27 dist-test-slave-sb6p krb5kdc[21297](info): commencing operation
krb5kdc: starting...
W20260924 03:01:28.694360 392 mini_kdc.cc:121] Time spent starting KDC: real 0.943s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:28 dist-test-slave-sb6p krb5kdc[21297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218888, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:01:28.805500 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35883
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35883
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:28.883831 21310 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:28.884032 21310 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:28.884053 21310 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:28.885180 21310 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:28.885221 21310 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:28.885233 21310 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:28.885244 21310 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:28.885255 21310 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:28.886724 21310 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35883
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35883
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21310
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:28.886945 21310 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:28.887145 21310 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:28.891707 21315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:28.891755 21316 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:28.891863 21310 server_base.cc:1034] running on GCE node
W20260924 03:01:28.891705 21318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:28.892122 21310 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:28.892359 21310 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:28.893509 21310 hybrid_clock.cc:648] HybridClock initialized: now 1790218888893488 us; error 34 us; skew 500 ppm
Sep 24 03:01:28 dist-test-slave-sb6p krb5kdc[21297](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:28.895887 21310 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:01:28.896270 21310 webserver.cc:460] Webserver started at http://127.0.0.1:38907/ using document root <none> and password file <none>
I20260924 03:01:28.896442 21310 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:28.896488 21310 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:28.896572 21310 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:28.897531 21310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "766c400132d147efabc40acfa1d5cf82"
format_stamp: "Formatted at 2026-09-24 03:01:28 on dist-test-slave-sb6p"
server_key: "c7848e055d3fb0c989d81cc6da6dd16d"
server_key_iv: "9e120a3f5943f3981d34d6ace59e2694"
server_key_version: "encryptionkey@0"
I20260924 03:01:28.897862 21310 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "766c400132d147efabc40acfa1d5cf82"
format_stamp: "Formatted at 2026-09-24 03:01:28 on dist-test-slave-sb6p"
server_key: "c7848e055d3fb0c989d81cc6da6dd16d"
server_key_iv: "9e120a3f5943f3981d34d6ace59e2694"
server_key_version: "encryptionkey@0"
I20260924 03:01:28.899058 21310 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:01:28.899700 21325 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:28.899865 21310 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:28.899948 21310 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "766c400132d147efabc40acfa1d5cf82"
format_stamp: "Formatted at 2026-09-24 03:01:28 on dist-test-slave-sb6p"
server_key: "c7848e055d3fb0c989d81cc6da6dd16d"
server_key_iv: "9e120a3f5943f3981d34d6ace59e2694"
server_key_version: "encryptionkey@0"
I20260924 03:01:28.900012 21310 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:28.908524 21310 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:28.910255 21310 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:28.910375 21310 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:28.913995 21310 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35883
I20260924 03:01:28.914040 21377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35883 every 8 connection(s)
I20260924 03:01:28.914328 21310 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:28.915167 21378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:28.917351 21378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82: Bootstrap starting.
I20260924 03:01:28.917889 21378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:28.918140 21378 log.cc:824] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:28.918816 21378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82: No bootstrap required, opened a new log
I20260924 03:01:28.919826 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 21310
I20260924 03:01:28.919914 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:28.920082 392 external_mini_cluster.cc:1428] Setting key edaea42f77159ae3a3f236ecf047fb47
I20260924 03:01:28.920455 21378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766c400132d147efabc40acfa1d5cf82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35883 } }
I20260924 03:01:28.920650 21378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:28.920724 21378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 766c400132d147efabc40acfa1d5cf82, State: Initialized, Role: FOLLOWER
I20260924 03:01:28.920856 21378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766c400132d147efabc40acfa1d5cf82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35883 } }
I20260924 03:01:28.920925 21378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:28.920948 21378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:28.920992 21378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:28.921898 21378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766c400132d147efabc40acfa1d5cf82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35883 } }
I20260924 03:01:28.921984 21378 leader_election.cc:302] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 766c400132d147efabc40acfa1d5cf82; no voters:
I20260924 03:01:28.922134 21378 leader_election.cc:288] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:28.922166 21383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:28.922264 21383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [term 1 LEADER]: Becoming Leader. State: Replica: 766c400132d147efabc40acfa1d5cf82, State: Running, Role: LEADER
I20260924 03:01:28.922390 21378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:28.922398 21383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766c400132d147efabc40acfa1d5cf82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35883 } }
I20260924 03:01:28.922750 21383 sys_catalog.cc:455] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 766c400132d147efabc40acfa1d5cf82. Latest consensus state: current_term: 1 leader_uuid: "766c400132d147efabc40acfa1d5cf82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766c400132d147efabc40acfa1d5cf82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35883 } } }
I20260924 03:01:28.922979 21383 sys_catalog.cc:458] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:28.923367 21392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:28.922901 21384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "766c400132d147efabc40acfa1d5cf82" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766c400132d147efabc40acfa1d5cf82" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35883 } } }
I20260924 03:01:28.923691 21384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:28.923944 21392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:28.925606 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:28.920583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:58021 (local address 127.0.0.1:35883)
0924 03:01:28.920765 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:28.920774 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:28.920807 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:01:28.920920 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:28.920926 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:28.920986 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:28.921123 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:28.921342 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:28.922364 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:28.923459 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:28.924039 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:28.924177 (+ 138us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:28.924900 (+ 723us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:28.924904 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:28.924904 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:28.925130 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:28.925167 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:28.925182 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:28.925269 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:28.925376 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 03:01:28.925405 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":76,"threads_started":1}
I20260924 03:01:28.925611 21392 catalog_manager.cc:1347] Generated new cluster ID: 0277a20cfe984306a46c5d0dd1ad59de
I20260924 03:01:28.925877 21392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:28.926512 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:28.934239 21382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:28.930787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:57587 (local address 127.0.0.1:35883)
0924 03:01:28.930825 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:28.930827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:28.930998 (+ 171us) server_negotiation.cc:407] Connection header received
0924 03:01:28.931029 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:28.931030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:28.931050 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:28.931123 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:28.931251 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:28.931990 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:28.932939 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:28.933261 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:28.933333 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:28.933991 (+ 658us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:28.933992 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:28.933993 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:28.933997 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:28.934015 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:28.934020 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:28.934086 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:28.934141 (+ 55us) server_negotiation.cc:299] Negotiation successful
0924 03:01:28.934155 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:01:28.935691 21392 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:28.936172 21392 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:28.943871 21392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 766c400132d147efabc40acfa1d5cf82: Generated new TSK 0
I20260924 03:01:28.944043 21392 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:28.949075 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 21310
W20260924 03:01:28.952893 21295 connection.cc:535] client connection to 127.0.0.1:35883 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35883): Connection reset by peer (error 104)
2026-09-24T03:01:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1214 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:28 dist-test-slave-sb6p krb5kdc[21409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:28 dist-test-slave-sb6p krb5kdc[21409](info): set up 2 sockets
Sep 24 03:01:28 dist-test-slave-sb6p krb5kdc[21409](info): commencing operation
krb5kdc: starting...
W20260924 03:01:29.894256 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:29 dist-test-slave-sb6p krb5kdc[21409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:01:30.005811 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35979
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38535
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35979
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:30.084664 21422 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:30.084883 21422 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:30.084915 21422 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:30.086093 21422 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:30.086136 21422 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:30.086155 21422 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:30.086171 21422 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:30.086187 21422 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:30.087669 21422 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38535
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21422
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:30.087909 21422 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:30.088110 21422 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:30.092674 21427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:30.092669 21428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:30.092757 21430 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:30.092837 21422 server_base.cc:1034] running on GCE node
I20260924 03:01:30.093129 21422 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:30.093390 21422 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:30.094543 21422 hybrid_clock.cc:648] HybridClock initialized: now 1790218890094514 us; error 40 us; skew 500 ppm
Sep 24 03:01:30 dist-test-slave-sb6p krb5kdc[21409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218890, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:30.096845 21422 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:01:30.097237 21422 webserver.cc:460] Webserver started at http://127.0.0.1:45239/ using document root <none> and password file <none>
I20260924 03:01:30.097426 21422 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:30.097472 21422 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:30.097595 21422 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:30.098676 21422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "752b8c5b729649798f12f534a501b4e3"
format_stamp: "Formatted at 2026-09-24 03:01:30 on dist-test-slave-sb6p"
server_key: "cf5368c62df0184142644781ecb5fcf2"
server_key_iv: "e5837340cb9283d36d41cb0d0f901b6d"
server_key_version: "encryptionkey@0"
I20260924 03:01:30.098995 21422 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "752b8c5b729649798f12f534a501b4e3"
format_stamp: "Formatted at 2026-09-24 03:01:30 on dist-test-slave-sb6p"
server_key: "cf5368c62df0184142644781ecb5fcf2"
server_key_iv: "e5837340cb9283d36d41cb0d0f901b6d"
server_key_version: "encryptionkey@0"
I20260924 03:01:30.100230 21422 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:01:30.100854 21437 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:30.101011 21422 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:01:30.101092 21422 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "752b8c5b729649798f12f534a501b4e3"
format_stamp: "Formatted at 2026-09-24 03:01:30 on dist-test-slave-sb6p"
server_key: "cf5368c62df0184142644781ecb5fcf2"
server_key_iv: "e5837340cb9283d36d41cb0d0f901b6d"
server_key_version: "encryptionkey@0"
I20260924 03:01:30.101152 21422 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:30.104303 21422 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:30.106004 21422 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:30.106127 21422 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:30.109779 21422 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35979
I20260924 03:01:30.109838 21489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35979 every 8 connection(s)
I20260924 03:01:30.110110 21422 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:30.111090 21490 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:30.113233 21490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3: Bootstrap starting.
I20260924 03:01:30.113785 21490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:30.114025 21490 log.cc:824] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:30.114707 21490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3: No bootstrap required, opened a new log
I20260924 03:01:30.116235 21490 raft_consensus.cc:348] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752b8c5b729649798f12f534a501b4e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35979 } }
I20260924 03:01:30.116374 21490 raft_consensus.cc:374] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:30.116395 21490 raft_consensus.cc:729] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 752b8c5b729649798f12f534a501b4e3, State: Initialized, Role: FOLLOWER
I20260924 03:01:30.116536 21490 consensus_queue.cc:260] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752b8c5b729649798f12f534a501b4e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35979 } }
I20260924 03:01:30.116614 21490 raft_consensus.cc:388] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:30.116638 21490 raft_consensus.cc:482] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:30.116684 21490 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:30.117184 21490 raft_consensus.cc:504] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752b8c5b729649798f12f534a501b4e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35979 } }
I20260924 03:01:30.117273 21490 leader_election.cc:302] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 752b8c5b729649798f12f534a501b4e3; no voters:
I20260924 03:01:30.117429 21490 leader_election.cc:288] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:30.117466 21493 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:30.117599 21493 raft_consensus.cc:686] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [term 1 LEADER]: Becoming Leader. State: Replica: 752b8c5b729649798f12f534a501b4e3, State: Running, Role: LEADER
I20260924 03:01:30.117676 21490 sys_catalog.cc:564] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:30.117686 21493 consensus_queue.cc:237] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752b8c5b729649798f12f534a501b4e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35979 } }
I20260924 03:01:30.118057 21494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "752b8c5b729649798f12f534a501b4e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752b8c5b729649798f12f534a501b4e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35979 } } }
I20260924 03:01:30.118197 21495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 752b8c5b729649798f12f534a501b4e3. Latest consensus state: current_term: 1 leader_uuid: "752b8c5b729649798f12f534a501b4e3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "752b8c5b729649798f12f534a501b4e3" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35979 } } }
I20260924 03:01:30.118283 21495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:30.118184 21494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:30.118628 21502 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:30.119068 21502 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:30.119887 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 21422
I20260924 03:01:30.119989 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:30.120159 392 external_mini_cluster.cc:1428] Setting key e57942ec07da326b684e6dabc69fd6d8
I20260924 03:01:30.121006 21502 catalog_manager.cc:1347] Generated new cluster ID: 185ed8414a3240fdbd919aacc924a209
I20260924 03:01:30.121048 21502 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:30.122169 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:30.120596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:37165 (local address 127.0.0.1:35979)
0924 03:01:30.120745 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:30.120754 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:30.120790 (+ 36us) server_negotiation.cc:407] Connection header received
0924 03:01:30.120903 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:30.120910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:30.120941 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:30.121100 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:30.121750 (+ 650us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:30.121754 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:30.121754 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:30.121758 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:30.121791 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:30.121804 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:30.121867 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:30.121929 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 03:01:30.121972 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260924 03:01:30.123437 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:30.131637 21513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:30.124795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:43369 (local address 127.0.0.1:35979)
0924 03:01:30.124826 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:30.124828 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:30.130723 (+ 5895us) server_negotiation.cc:407] Connection header received
0924 03:01:30.130782 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:30.130783 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:30.130789 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:30.130850 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:30.131430 (+ 580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:30.131431 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:30.131431 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:30.131435 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:30.131449 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:30.131453 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:30.131498 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:30.131551 (+ 53us) server_negotiation.cc:299] Negotiation successful
0924 03:01:30.131564 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:01:30.134853 21502 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:30.135010 21502 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:30.140411 21502 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 752b8c5b729649798f12f534a501b4e3: Generated new TSK 0
I20260924 03:01:30.140581 21502 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:30.147564 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 21422
2026-09-24T03:01:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1198 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:30 dist-test-slave-sb6p krb5kdc[21521](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:30 dist-test-slave-sb6p krb5kdc[21521](info): set up 2 sockets
Sep 24 03:01:30 dist-test-slave-sb6p krb5kdc[21521](info): commencing operation
krb5kdc: starting...
W20260924 03:01:31.098348 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:31 dist-test-slave-sb6p krb5kdc[21521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:01:31.209587 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32769
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:42111
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:32769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:31.288760 21534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:31.288956 21534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:31.288977 21534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:31.290076 21534 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:31.290108 21534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:31.290122 21534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:31.290132 21534 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:31.290149 21534 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:31.291572 21534 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42111
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:32769
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:32769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21534
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:31.291752 21534 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:31.291914 21534 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:31.296343 21542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:31.296350 21540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:31.296481 21539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:31.296514 21534 server_base.cc:1034] running on GCE node
I20260924 03:01:31.296814 21534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:31.297055 21534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:31.298190 21534 hybrid_clock.cc:648] HybridClock initialized: now 1790218891298186 us; error 38 us; skew 500 ppm
Sep 24 03:01:31 dist-test-slave-sb6p krb5kdc[21521](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218891, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:31.300437 21534 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:01:31.300866 21534 webserver.cc:460] Webserver started at http://127.0.0.1:35413/ using document root <none> and password file <none>
I20260924 03:01:31.301076 21534 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:31.301122 21534 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:31.301244 21534 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:31.302240 21534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "4d02fa9ac261499b80d48bc94fd6b2b5"
format_stamp: "Formatted at 2026-09-24 03:01:31 on dist-test-slave-sb6p"
server_key: "f4d109452590740a797561e052a646fb"
server_key_iv: "7a52b76afea5b06a0e06d45ae7a17cb4"
server_key_version: "encryptionkey@0"
I20260924 03:01:31.302598 21534 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "4d02fa9ac261499b80d48bc94fd6b2b5"
format_stamp: "Formatted at 2026-09-24 03:01:31 on dist-test-slave-sb6p"
server_key: "f4d109452590740a797561e052a646fb"
server_key_iv: "7a52b76afea5b06a0e06d45ae7a17cb4"
server_key_version: "encryptionkey@0"
I20260924 03:01:31.303795 21534 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:01:31.304443 21549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:31.304621 21534 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:01:31.304710 21534 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "4d02fa9ac261499b80d48bc94fd6b2b5"
format_stamp: "Formatted at 2026-09-24 03:01:31 on dist-test-slave-sb6p"
server_key: "f4d109452590740a797561e052a646fb"
server_key_iv: "7a52b76afea5b06a0e06d45ae7a17cb4"
server_key_version: "encryptionkey@0"
I20260924 03:01:31.304766 21534 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:31.311751 21534 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:31.313360 21534 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:31.313459 21534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:31.317219 21534 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:32769
I20260924 03:01:31.317246 21601 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:32769 every 8 connection(s)
I20260924 03:01:31.317561 21534 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:31.318342 21602 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:31.320564 21602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5: Bootstrap starting.
I20260924 03:01:31.321097 21602 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:31.321354 21602 log.cc:824] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:31.321974 21602 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5: No bootstrap required, opened a new log
I20260924 03:01:31.323593 21602 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02fa9ac261499b80d48bc94fd6b2b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } }
I20260924 03:01:31.323791 21602 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:31.323815 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 21534
I20260924 03:01:31.323827 21602 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d02fa9ac261499b80d48bc94fd6b2b5, State: Initialized, Role: FOLLOWER
I20260924 03:01:31.323922 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:31.323972 21602 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02fa9ac261499b80d48bc94fd6b2b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } }
I20260924 03:01:31.324049 21602 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:31.324084 21602 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:31.324121 392 external_mini_cluster.cc:1428] Setting key defb236f0fba5e20535f4bca788c6cd1
I20260924 03:01:31.324131 21602 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:31.324729 21602 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02fa9ac261499b80d48bc94fd6b2b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } }
I20260924 03:01:31.324828 21602 leader_election.cc:302] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d02fa9ac261499b80d48bc94fd6b2b5; no voters:
I20260924 03:01:31.324980 21602 leader_election.cc:288] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:31.325022 21607 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:31.325224 21607 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [term 1 LEADER]: Becoming Leader. State: Replica: 4d02fa9ac261499b80d48bc94fd6b2b5, State: Running, Role: LEADER
I20260924 03:01:31.325223 21602 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:31.325408 21607 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02fa9ac261499b80d48bc94fd6b2b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } }
I20260924 03:01:31.325798 21609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d02fa9ac261499b80d48bc94fd6b2b5. Latest consensus state: current_term: 1 leader_uuid: "4d02fa9ac261499b80d48bc94fd6b2b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02fa9ac261499b80d48bc94fd6b2b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } } }
I20260924 03:01:31.325883 21609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:31.326035 21608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d02fa9ac261499b80d48bc94fd6b2b5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d02fa9ac261499b80d48bc94fd6b2b5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 32769 } } }
I20260924 03:01:31.326071 21613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:31.326103 21608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:31.326582 21613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:31.328094 21613 catalog_manager.cc:1347] Generated new cluster ID: e835afd4e98c4799930b63f1e8413970
I20260924 03:01:31.328143 21613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:01:31 dist-test-slave-sb6p krb5kdc[21521](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218891, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260924 03:01:31.333849 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:31.324598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:35325 (local address 127.0.0.1:32769)
0924 03:01:31.324770 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:31.324777 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:31.324808 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:01:31.324925 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:31.324931 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:31.324985 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:31.325132 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:31.325819 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:31.327200 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:31.328100 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:31.328569 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:31.328665 (+ 96us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:31.330947 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:31.330951 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:31.330951 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:31.330961 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:31.332424 (+ 1463us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:31.332789 (+ 365us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:31.332791 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:31.332792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:31.332882 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:31.333083 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:31.333084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:31.333084 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:31.333306 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:31.333458 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:31.333575 (+ 117us) server_negotiation.cc:299] Negotiation successful
0924 03:01:31.333680 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":85,"threads_started":1}
I20260924 03:01:31.334834 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:31.338407 21613 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:31.339380 21613 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:31.341871 21606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:31.336201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:34729 (local address 127.0.0.1:32769)
0924 03:01:31.336233 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:31.336235 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:31.336372 (+ 137us) server_negotiation.cc:407] Connection header received
0924 03:01:31.336436 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:31.336436 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:31.336446 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:31.336481 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:31.336974 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:31.337442 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:31.338180 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:31.338437 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:31.338495 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:31.339664 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:31.339665 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:31.339665 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:31.339672 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:31.340857 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:31.341188 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:31.341189 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:31.341189 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:31.341240 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:31.341425 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:31.341425 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:31.341426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:31.341523 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:31.341623 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:31.341735 (+ 112us) server_negotiation.cc:299] Negotiation successful
0924 03:01:31.341791 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260924 03:01:31.346455 21613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d02fa9ac261499b80d48bc94fd6b2b5: Generated new TSK 0
I20260924 03:01:31.346649 21613 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:31.350483 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 21534
W20260924 03:01:31.354188 21519 connection.cc:535] client connection to 127.0.0.1:32769 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:32769): Connection reset by peer (error 104)
2026-09-24T03:01:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1202 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:31 dist-test-slave-sb6p krb5kdc[21633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:31 dist-test-slave-sb6p krb5kdc[21633](info): set up 2 sockets
Sep 24 03:01:31 dist-test-slave-sb6p krb5kdc[21633](info): commencing operation
krb5kdc: starting...
W20260924 03:01:32.306248 392 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:32 dist-test-slave-sb6p krb5kdc[21633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:01:32.414815 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45809
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44279
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:32.493019 21646 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:32.493228 21646 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:32.493259 21646 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:32.494392 21646 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:32.494433 21646 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:32.494452 21646 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:32.494468 21646 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:32.494483 21646 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:32.495944 21646 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44279
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45809
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21646
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:32.496176 21646 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:32.496384 21646 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:32.501057 21654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:32.501081 21652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:32.501087 21651 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:32.501164 21646 server_base.cc:1034] running on GCE node
I20260924 03:01:32.501549 21646 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:32.501821 21646 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:32.502990 21646 hybrid_clock.cc:648] HybridClock initialized: now 1790218892502963 us; error 45 us; skew 500 ppm
Sep 24 03:01:32 dist-test-slave-sb6p krb5kdc[21633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218892, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:32.505300 21646 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:01:32.505764 21646 webserver.cc:460] Webserver started at http://127.0.0.1:43583/ using document root <none> and password file <none>
I20260924 03:01:32.505963 21646 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:32.506011 21646 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:32.506134 21646 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:32.507201 21646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "e22f2014d1ae43b29908090cbae8c503"
format_stamp: "Formatted at 2026-09-24 03:01:32 on dist-test-slave-sb6p"
server_key: "da53b9ecd731c5f1da4937972f0a9025"
server_key_iv: "bb3bafd4fee8993718c7210f763d513b"
server_key_version: "encryptionkey@0"
I20260924 03:01:32.507519 21646 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "e22f2014d1ae43b29908090cbae8c503"
format_stamp: "Formatted at 2026-09-24 03:01:32 on dist-test-slave-sb6p"
server_key: "da53b9ecd731c5f1da4937972f0a9025"
server_key_iv: "bb3bafd4fee8993718c7210f763d513b"
server_key_version: "encryptionkey@0"
I20260924 03:01:32.508809 21646 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:01:32.509480 21661 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:32.509641 21646 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:32.509723 21646 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "e22f2014d1ae43b29908090cbae8c503"
format_stamp: "Formatted at 2026-09-24 03:01:32 on dist-test-slave-sb6p"
server_key: "da53b9ecd731c5f1da4937972f0a9025"
server_key_iv: "bb3bafd4fee8993718c7210f763d513b"
server_key_version: "encryptionkey@0"
I20260924 03:01:32.509783 21646 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:32.535871 21646 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:32.537660 21646 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:32.537798 21646 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:32.541729 21646 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45809
I20260924 03:01:32.541779 21713 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45809 every 8 connection(s)
I20260924 03:01:32.542117 21646 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:32.543092 21714 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:32.545570 21714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503: Bootstrap starting.
I20260924 03:01:32.546152 21714 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:32.546406 21714 log.cc:824] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:32.547127 21714 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503: No bootstrap required, opened a new log
I20260924 03:01:32.548722 21714 raft_consensus.cc:348] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f2014d1ae43b29908090cbae8c503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } }
I20260924 03:01:32.548862 21714 raft_consensus.cc:374] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:32.548884 21714 raft_consensus.cc:729] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e22f2014d1ae43b29908090cbae8c503, State: Initialized, Role: FOLLOWER
I20260924 03:01:32.549031 21714 consensus_queue.cc:260] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f2014d1ae43b29908090cbae8c503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } }
I20260924 03:01:32.549101 21714 raft_consensus.cc:388] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:32.549132 21714 raft_consensus.cc:482] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:32.549180 21714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:32.549265 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 21646
I20260924 03:01:32.549369 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:32.549562 392 external_mini_cluster.cc:1428] Setting key f07993c6fd1befdbf0631dbd0520ba0f
I20260924 03:01:32.549829 21714 raft_consensus.cc:504] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f2014d1ae43b29908090cbae8c503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } }
I20260924 03:01:32.549927 21714 leader_election.cc:302] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e22f2014d1ae43b29908090cbae8c503; no voters:
I20260924 03:01:32.550094 21714 leader_election.cc:288] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:32.550177 21718 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:32.550285 21718 raft_consensus.cc:686] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [term 1 LEADER]: Becoming Leader. State: Replica: e22f2014d1ae43b29908090cbae8c503, State: Running, Role: LEADER
I20260924 03:01:32.550400 21714 sys_catalog.cc:564] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:32.550418 21718 consensus_queue.cc:237] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f2014d1ae43b29908090cbae8c503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } }
I20260924 03:01:32.550897 21718 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e22f2014d1ae43b29908090cbae8c503. Latest consensus state: current_term: 1 leader_uuid: "e22f2014d1ae43b29908090cbae8c503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f2014d1ae43b29908090cbae8c503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } } }
I20260924 03:01:32.550936 21721 sys_catalog.cc:455] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e22f2014d1ae43b29908090cbae8c503" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e22f2014d1ae43b29908090cbae8c503" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45809 } } }
I20260924 03:01:32.551024 21718 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:32.551036 21721 sys_catalog.cc:458] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:32.551287 21728 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:32.551858 21728 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:32.553370 21728 catalog_manager.cc:1347] Generated new cluster ID: 99e421f6d82b445d8519ccea6495d64d
I20260924 03:01:32.553426 21728 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:32.555796 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:32.549997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:36325 (local address 127.0.0.1:45809)
0924 03:01:32.550184 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:32.550194 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:32.550228 (+ 34us) server_negotiation.cc:407] Connection header received
0924 03:01:32.550348 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:32.550355 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:32.550415 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:32.550562 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:32.550775 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:32.552485 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:32.553467 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:32.554061 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:32.554151 (+ 90us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:32.555021 (+ 870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:32.555024 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:32.555025 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:32.555221 (+ 196us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:32.555258 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:32.555272 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:32.555338 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:32.555557 (+ 219us) server_negotiation.cc:299] Negotiation successful
0924 03:01:32.555582 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":70,"threads_started":1}
I20260924 03:01:32.556535 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:32.560156 21728 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:32.560820 21728 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:32.561372 21719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:32.557923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:45709 (local address 127.0.0.1:45809)
0924 03:01:32.557955 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:32.557957 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:32.558053 (+ 96us) server_negotiation.cc:407] Connection header received
0924 03:01:32.558132 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:32.558132 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:32.558142 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:32.558166 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:32.558341 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:32.559161 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:32.560061 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:32.560333 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:32.560375 (+ 42us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:32.561134 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:32.561135 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:32.561136 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:32.561140 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:32.561153 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:32.561157 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:32.561183 (+ 26us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:32.561290 (+ 107us) server_negotiation.cc:299] Negotiation successful
0924 03:01:32.561301 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":45,"server-negotiator.queue_time_us":17}
I20260924 03:01:32.567898 21728 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e22f2014d1ae43b29908090cbae8c503: Generated new TSK 0
I20260924 03:01:32.568078 21728 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:32.572054 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 21646
W20260924 03:01:32.575685 21631 connection.cc:535] client connection to 127.0.0.1:45809 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45809): Connection reset by peer (error 104)
2026-09-24T03:01:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:32 dist-test-slave-sb6p krb5kdc[21745](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:32 dist-test-slave-sb6p krb5kdc[21745](info): set up 2 sockets
Sep 24 03:01:32 dist-test-slave-sb6p krb5kdc[21745](info): commencing operation
krb5kdc: starting...
W20260924 03:01:33.515328 392 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:33 dist-test-slave-sb6p krb5kdc[21745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218893, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260924 03:01:33.629292 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42777
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42777
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:33.707377 21758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:33.707585 21758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:33.707615 21758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:33.708746 21758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:33.708784 21758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:33.708803 21758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:33.708819 21758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:33.708834 21758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:33.710248 21758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42777
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21758
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:33.710475 21758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:33.710693 21758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:33.715344 21763 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:33.715351 21764 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:33.715381 21766 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:33.715525 21758 server_base.cc:1034] running on GCE node
I20260924 03:01:33.715780 21758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:33.716044 21758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:33.717204 21758 hybrid_clock.cc:648] HybridClock initialized: now 1790218893717189 us; error 28 us; skew 500 ppm
Sep 24 03:01:33 dist-test-slave-sb6p krb5kdc[21745](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218893, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:33.719561 21758 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260924 03:01:33.719969 21758 webserver.cc:460] Webserver started at http://127.0.0.1:44999/ using document root <none> and password file <none>
I20260924 03:01:33.720147 21758 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:33.720185 21758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:33.720273 21758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:33.721233 21758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "28b2812df4614710abeb39800fe590f7"
format_stamp: "Formatted at 2026-09-24 03:01:33 on dist-test-slave-sb6p"
server_key: "6870ebf40e92295a181da8533e1f6e57"
server_key_iv: "d054fff623317e428858e540a58dd1fb"
server_key_version: "encryptionkey@0"
I20260924 03:01:33.721511 21758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "28b2812df4614710abeb39800fe590f7"
format_stamp: "Formatted at 2026-09-24 03:01:33 on dist-test-slave-sb6p"
server_key: "6870ebf40e92295a181da8533e1f6e57"
server_key_iv: "d054fff623317e428858e540a58dd1fb"
server_key_version: "encryptionkey@0"
I20260924 03:01:33.722719 21758 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:01:33.723421 21773 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:33.723583 21758 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:33.723666 21758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "28b2812df4614710abeb39800fe590f7"
format_stamp: "Formatted at 2026-09-24 03:01:33 on dist-test-slave-sb6p"
server_key: "6870ebf40e92295a181da8533e1f6e57"
server_key_iv: "d054fff623317e428858e540a58dd1fb"
server_key_version: "encryptionkey@0"
I20260924 03:01:33.723738 21758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:33.726794 21758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:33.728500 21758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:33.728607 21758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:33.732244 21758 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42777
I20260924 03:01:33.732276 21825 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42777 every 8 connection(s)
I20260924 03:01:33.732553 21758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:33.733378 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 21758
I20260924 03:01:33.733548 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:33.733561 21826 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:33.733764 392 external_mini_cluster.cc:1428] Setting key 425ac1de24b80370323782791435447d
I20260924 03:01:33.736095 21826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7: Bootstrap starting.
I20260924 03:01:33.736310 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:33.734255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:43237 (local address 127.0.0.1:42777)
0924 03:01:33.734638 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:33.734645 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:33.734676 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:01:33.735074 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:33.735080 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:33.735103 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:33.735236 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:33.735867 (+ 631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:33.735870 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:33.735871 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:33.735874 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:33.735913 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:33.735922 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:33.735976 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:33.736044 (+ 68us) server_negotiation.cc:299] Negotiation successful
0924 03:01:33.736086 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":67,"threads_started":1}
I20260924 03:01:33.736755 21826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:33.737062 21826 log.cc:824] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:33.737205 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:33.737762 21826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7: No bootstrap required, opened a new log
I20260924 03:01:33.739549 21829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:33.738617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.32:34497 (local address 127.0.0.1:42777)
0924 03:01:33.738661 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:33.738663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:33.738771 (+ 108us) server_negotiation.cc:407] Connection header received
0924 03:01:33.738823 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:33.738824 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:33.738829 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:33.738872 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:33.739359 (+ 487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:33.739360 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:33.739361 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:33.739365 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:33.739376 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:33.739379 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:33.739430 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:33.739481 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 03:01:33.739493 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260924 03:01:33.739935 21826 raft_consensus.cc:348] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b2812df4614710abeb39800fe590f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } }
I20260924 03:01:33.740115 21826 raft_consensus.cc:374] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:33.740145 21826 raft_consensus.cc:729] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 28b2812df4614710abeb39800fe590f7, State: Initialized, Role: FOLLOWER
I20260924 03:01:33.740262 21826 consensus_queue.cc:260] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b2812df4614710abeb39800fe590f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } }
I20260924 03:01:33.740334 21826 raft_consensus.cc:388] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:33.740368 21826 raft_consensus.cc:482] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:33.740411 21826 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:33.740916 21826 raft_consensus.cc:504] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b2812df4614710abeb39800fe590f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } }
I20260924 03:01:33.741011 21826 leader_election.cc:302] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 28b2812df4614710abeb39800fe590f7; no voters:
I20260924 03:01:33.741169 21826 leader_election.cc:288] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:33.741214 21836 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:33.741326 21836 raft_consensus.cc:686] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [term 1 LEADER]: Becoming Leader. State: Replica: 28b2812df4614710abeb39800fe590f7, State: Running, Role: LEADER
I20260924 03:01:33.741400 21836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b2812df4614710abeb39800fe590f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } }
I20260924 03:01:33.741400 21826 sys_catalog.cc:564] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:33.741732 21838 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "28b2812df4614710abeb39800fe590f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b2812df4614710abeb39800fe590f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } } }
I20260924 03:01:33.741739 21837 sys_catalog.cc:455] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 28b2812df4614710abeb39800fe590f7. Latest consensus state: current_term: 1 leader_uuid: "28b2812df4614710abeb39800fe590f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "28b2812df4614710abeb39800fe590f7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42777 } } }
I20260924 03:01:33.741839 21837 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:33.741829 21838 sys_catalog.cc:458] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:33.742110 21843 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:33.742903 21843 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:33.744336 21843 catalog_manager.cc:1347] Generated new cluster ID: f2130926a5b04706b7107e2e6cbd84ae
I20260924 03:01:33.744385 21843 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:33.751605 21843 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:33.751753 21843 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:33.757961 21843 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 28b2812df4614710abeb39800fe590f7: Generated new TSK 0
I20260924 03:01:33.758134 21843 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:33.763007 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 21758
2026-09-24T03:01:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1190 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:33 dist-test-slave-sb6p krb5kdc[21857](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:33 dist-test-slave-sb6p krb5kdc[21857](info): set up 2 sockets
Sep 24 03:01:33 dist-test-slave-sb6p krb5kdc[21857](info): commencing operation
krb5kdc: starting...
W20260924 03:01:34.706331 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:34 dist-test-slave-sb6p krb5kdc[21857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:01:34.817184 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36433
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:36433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:34.895814 21870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:34.896018 21870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:34.896046 21870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:34.897163 21870 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:34.897208 21870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:34.897226 21870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:34.897243 21870 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:34.897258 21870 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:34.898720 21870 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:36433
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:36433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:34.898957 21870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:34.899148 21870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:34.903594 21878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:34.903649 21870 server_base.cc:1034] running on GCE node
W20260924 03:01:34.903553 21876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:34.903769 21875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:34.903986 21870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:34.904237 21870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:34.905368 21870 hybrid_clock.cc:648] HybridClock initialized: now 1790218894905355 us; error 27 us; skew 500 ppm
Sep 24 03:01:34 dist-test-slave-sb6p krb5kdc[21857](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218894, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:34.907648 21870 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:01:34.908069 21870 webserver.cc:460] Webserver started at http://127.0.98.62:44913/ using document root <none> and password file <none>
I20260924 03:01:34.908272 21870 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:34.908317 21870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:34.908432 21870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:34.909435 21870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "44df0270afd04b11889a1ba121bcfc8c"
format_stamp: "Formatted at 2026-09-24 03:01:34 on dist-test-slave-sb6p"
server_key: "8a2461b23572f99e69c9524a7bf3ecde"
server_key_iv: "93300642873e4e5119a265cc9e45b64e"
server_key_version: "encryptionkey@0"
I20260924 03:01:34.909749 21870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "44df0270afd04b11889a1ba121bcfc8c"
format_stamp: "Formatted at 2026-09-24 03:01:34 on dist-test-slave-sb6p"
server_key: "8a2461b23572f99e69c9524a7bf3ecde"
server_key_iv: "93300642873e4e5119a265cc9e45b64e"
server_key_version: "encryptionkey@0"
I20260924 03:01:34.910977 21870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:01:34.911615 21885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:34.911756 21870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:34.911834 21870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "44df0270afd04b11889a1ba121bcfc8c"
format_stamp: "Formatted at 2026-09-24 03:01:34 on dist-test-slave-sb6p"
server_key: "8a2461b23572f99e69c9524a7bf3ecde"
server_key_iv: "93300642873e4e5119a265cc9e45b64e"
server_key_version: "encryptionkey@0"
I20260924 03:01:34.911891 21870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:34.926970 21870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:34.929047 21870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:34.929189 21870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:34.933038 21870 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:36433
I20260924 03:01:34.933085 21937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:36433 every 8 connection(s)
I20260924 03:01:34.933413 21870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:34.934218 21938 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:34.936443 21938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c: Bootstrap starting.
I20260924 03:01:34.936968 21938 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:34.937217 21938 log.cc:824] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:34.937821 21938 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c: No bootstrap required, opened a new log
I20260924 03:01:34.939443 21938 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44df0270afd04b11889a1ba121bcfc8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36433 } }
I20260924 03:01:34.939580 21938 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:34.939602 21938 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44df0270afd04b11889a1ba121bcfc8c, State: Initialized, Role: FOLLOWER
I20260924 03:01:34.939733 21938 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44df0270afd04b11889a1ba121bcfc8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36433 } }
I20260924 03:01:34.939790 21938 raft_consensus.cc:388] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:34.939822 21938 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:34.939867 21938 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:34.940352 21938 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44df0270afd04b11889a1ba121bcfc8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36433 } }
I20260924 03:01:34.940438 21938 leader_election.cc:302] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 44df0270afd04b11889a1ba121bcfc8c; no voters:
I20260924 03:01:34.940603 21938 leader_election.cc:288] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:34.940632 21941 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:34.940821 21941 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [term 1 LEADER]: Becoming Leader. State: Replica: 44df0270afd04b11889a1ba121bcfc8c, State: Running, Role: LEADER
I20260924 03:01:34.940819 21938 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:34.940937 21941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44df0270afd04b11889a1ba121bcfc8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36433 } }
I20260924 03:01:34.941437 21942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "44df0270afd04b11889a1ba121bcfc8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44df0270afd04b11889a1ba121bcfc8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36433 } } }
I20260924 03:01:34.941520 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 21870
I20260924 03:01:34.941526 21942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:34.941627 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:34.941684 21943 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44df0270afd04b11889a1ba121bcfc8c. Latest consensus state: current_term: 1 leader_uuid: "44df0270afd04b11889a1ba121bcfc8c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "44df0270afd04b11889a1ba121bcfc8c" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 36433 } } }
I20260924 03:01:34.941749 21943 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:34.941802 392 external_mini_cluster.cc:1428] Setting key a00e4b981f58d3b443e3786051d9c6f4
I20260924 03:01:34.942028 21950 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:34.942641 21950 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:34.943985 21950 catalog_manager.cc:1347] Generated new cluster ID: 8ccfc1e897d5420290ccf8b31ee9bd64
I20260924 03:01:34.944033 21950 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:01:34 dist-test-slave-sb6p krb5kdc[21857](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218894, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:34.949100 21950 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:34.949681 21950 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:34.952911 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:34.942397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57125 (local address 127.0.98.62:36433)
0924 03:01:34.942616 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:34.942626 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:34.942657 (+ 31us) server_negotiation.cc:407] Connection header received
0924 03:01:34.942771 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:34.942778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:34.942832 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:34.942971 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:34.943720 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:34.945075 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:34.946192 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:34.946793 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:34.946884 (+ 91us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:34.949465 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:34.949469 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:34.949469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:34.949481 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:34.951375 (+ 1894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:34.951779 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:34.951782 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:34.951782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:34.951882 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:34.952081 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:34.952082 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:34.952082 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:34.952355 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:34.952515 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:34.952594 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 03:01:34.952691 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":172,"thread_start_us":86,"threads_started":1}
I20260924 03:01:34.953351 21950 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44df0270afd04b11889a1ba121bcfc8c: Generated new TSK 0
I20260924 03:01:34.953488 21950 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:34.953814 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:34.961560 21960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:34.955335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35171 (local address 127.0.98.62:36433)
0924 03:01:34.955372 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:34.955373 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:34.955423 (+ 50us) server_negotiation.cc:407] Connection header received
0924 03:01:34.955471 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:34.955472 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:34.955481 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:34.955538 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:34.955977 (+ 439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:34.956778 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:34.957451 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:34.957768 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:34.957828 (+ 60us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:34.959042 (+ 1214us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:34.959043 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:34.959043 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:34.959049 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:34.960481 (+ 1432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:34.960871 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:34.960872 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:34.960873 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:34.960936 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:34.961114 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:34.961115 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:34.961115 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:34.961238 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:34.961333 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:34.961408 (+ 75us) server_negotiation.cc:299] Negotiation successful
0924 03:01:34.961470 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:01:34.963238 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 21870
W20260924 03:01:34.967008 21855 connection.cc:535] client connection to 127.0.98.62:36433 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:36433): Connection reset by peer (error 104)
2026-09-24T03:01:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1200 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:35 dist-test-slave-sb6p krb5kdc[21969](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:35 dist-test-slave-sb6p krb5kdc[21969](info): set up 2 sockets
Sep 24 03:01:35 dist-test-slave-sb6p krb5kdc[21969](info): commencing operation
krb5kdc: starting...
W20260924 03:01:35.914309 392 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.003s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:35 dist-test-slave-sb6p krb5kdc[21969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:01:36.021382 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40037
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34711
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:40037
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:36.098763 21982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:36.098965 21982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:36.098994 21982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:36.100142 21982 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:36.100184 21982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:36.100203 21982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:36.100219 21982 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:36.100234 21982 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:36.101670 21982 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34711
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:40037
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:40037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:36.101907 21982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:36.102106 21982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:36.106771 21988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:36.106835 21987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:36.106838 21990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:36.107012 21982 server_base.cc:1034] running on GCE node
I20260924 03:01:36.107240 21982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:36.107493 21982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:36.108625 21982 hybrid_clock.cc:648] HybridClock initialized: now 1790218896108615 us; error 31 us; skew 500 ppm
Sep 24 03:01:36 dist-test-slave-sb6p krb5kdc[21969](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218896, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:36.110976 21982 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:01:36.111377 21982 webserver.cc:460] Webserver started at http://127.0.98.62:44613/ using document root <none> and password file <none>
I20260924 03:01:36.111570 21982 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:36.111615 21982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:36.111742 21982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:36.112753 21982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "f1871bea2ba942ad8fe920c390b499c2"
format_stamp: "Formatted at 2026-09-24 03:01:36 on dist-test-slave-sb6p"
server_key: "ff5f4a9592784eda0dcfca0cb9e46546"
server_key_iv: "078f61fa70bd03faa1355586a5df7121"
server_key_version: "encryptionkey@0"
I20260924 03:01:36.113068 21982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "f1871bea2ba942ad8fe920c390b499c2"
format_stamp: "Formatted at 2026-09-24 03:01:36 on dist-test-slave-sb6p"
server_key: "ff5f4a9592784eda0dcfca0cb9e46546"
server_key_iv: "078f61fa70bd03faa1355586a5df7121"
server_key_version: "encryptionkey@0"
I20260924 03:01:36.114235 21982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:01:36.114912 21997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:36.115051 21982 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:36.115129 21982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "f1871bea2ba942ad8fe920c390b499c2"
format_stamp: "Formatted at 2026-09-24 03:01:36 on dist-test-slave-sb6p"
server_key: "ff5f4a9592784eda0dcfca0cb9e46546"
server_key_iv: "078f61fa70bd03faa1355586a5df7121"
server_key_version: "encryptionkey@0"
I20260924 03:01:36.115211 21982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:36.134469 21982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:36.136178 21982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:36.136309 21982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:36.139942 21982 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:40037
I20260924 03:01:36.139987 22049 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:40037 every 8 connection(s)
I20260924 03:01:36.140295 21982 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:36.141086 22050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:36.143280 22050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2: Bootstrap starting.
I20260924 03:01:36.143826 22050 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:36.144070 22050 log.cc:824] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:36.144690 22050 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2: No bootstrap required, opened a new log
I20260924 03:01:36.145720 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 21982
I20260924 03:01:36.145850 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:36.146040 392 external_mini_cluster.cc:1428] Setting key d57560bfb85264f027e5e02693ce4f6c
I20260924 03:01:36.146497 22050 raft_consensus.cc:348] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1871bea2ba942ad8fe920c390b499c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } }
I20260924 03:01:36.146708 22050 raft_consensus.cc:374] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:36.146736 22050 raft_consensus.cc:729] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f1871bea2ba942ad8fe920c390b499c2, State: Initialized, Role: FOLLOWER
I20260924 03:01:36.146858 22050 consensus_queue.cc:260] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1871bea2ba942ad8fe920c390b499c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } }
I20260924 03:01:36.146926 22050 raft_consensus.cc:388] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:36.146948 22050 raft_consensus.cc:482] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:36.146992 22050 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:36.147876 22050 raft_consensus.cc:504] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1871bea2ba942ad8fe920c390b499c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } }
I20260924 03:01:36.147961 22050 leader_election.cc:302] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f1871bea2ba942ad8fe920c390b499c2; no voters:
I20260924 03:01:36.148106 22050 leader_election.cc:288] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:36.148145 22055 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:36.148252 22055 raft_consensus.cc:686] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [term 1 LEADER]: Becoming Leader. State: Replica: f1871bea2ba942ad8fe920c390b499c2, State: Running, Role: LEADER
I20260924 03:01:36.148350 22050 sys_catalog.cc:564] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:36.148386 22055 consensus_queue.cc:237] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1871bea2ba942ad8fe920c390b499c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } }
I20260924 03:01:36.148725 22055 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f1871bea2ba942ad8fe920c390b499c2. Latest consensus state: current_term: 1 leader_uuid: "f1871bea2ba942ad8fe920c390b499c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1871bea2ba942ad8fe920c390b499c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } } }
I20260924 03:01:36.148736 22056 sys_catalog.cc:455] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f1871bea2ba942ad8fe920c390b499c2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f1871bea2ba942ad8fe920c390b499c2" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 40037 } } }
I20260924 03:01:36.148802 22055 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:36.148849 22056 sys_catalog.cc:458] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:36.149067 22062 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:36.149547 22062 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:36.150995 22062 catalog_manager.cc:1347] Generated new cluster ID: c006ddea18574ff685b5397b15c7acb4
I20260924 03:01:36.151046 22062 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:36.151364 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:36.146559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52353 (local address 127.0.98.62:40037)
0924 03:01:36.146733 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:36.146743 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:36.146776 (+ 33us) server_negotiation.cc:407] Connection header received
0924 03:01:36.146886 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:36.146891 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:36.146937 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:36.147069 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:36.147276 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:36.148209 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:36.149286 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:36.149948 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:36.150042 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:36.150704 (+ 662us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:36.150707 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:36.150708 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:36.150865 (+ 157us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:36.150895 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:36.150909 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:36.151025 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:36.151171 (+ 146us) server_negotiation.cc:299] Negotiation successful
0924 03:01:36.151200 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":71,"threads_started":1}
I20260924 03:01:36.152174 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:36.157495 22054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:36.153343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57513 (local address 127.0.98.62:40037)
0924 03:01:36.153375 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:36.153377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:36.154735 (+ 1358us) server_negotiation.cc:407] Connection header received
0924 03:01:36.154786 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:36.154786 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:36.154798 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:36.154844 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:36.154960 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:36.155420 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:36.156062 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:36.156628 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:36.156685 (+ 57us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:36.157281 (+ 596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:36.157282 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:36.157282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:36.157286 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:36.157299 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:36.157303 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:36.157347 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:36.157413 (+ 66us) server_negotiation.cc:299] Negotiation successful
0924 03:01:36.157425 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260924 03:01:36.202105 22062 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:36.202697 22062 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:36.207579 22062 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f1871bea2ba942ad8fe920c390b499c2: Generated new TSK 0
I20260924 03:01:36.207755 22062 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:36.214154 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 21982
W20260924 03:01:36.217974 21967 connection.cc:535] client connection to 127.0.98.62:40037 recv error: Network error: failed to read from TLS socket (remote: 127.0.98.62:40037): Connection reset by peer (error 104)
2026-09-24T03:01:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1251 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:36 dist-test-slave-sb6p krb5kdc[22081](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:36 dist-test-slave-sb6p krb5kdc[22081](info): set up 2 sockets
Sep 24 03:01:36 dist-test-slave-sb6p krb5kdc[22081](info): commencing operation
krb5kdc: starting...
W20260924 03:01:37.158303 392 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:37 dist-test-slave-sb6p krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218897, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:01:37.268599 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39125
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:36431
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:39125
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:37.346601 22094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:37.346804 22094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:37.346835 22094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:37.347970 22094 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:37.348011 22094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:37.348028 22094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:37.348044 22094 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:37.348059 22094 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:37.349485 22094 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:36431
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:39125
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:39125
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:37.349730 22094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:37.349923 22094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:37.354570 22099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:37.354753 22100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:37.354789 22102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:37.354851 22094 server_base.cc:1034] running on GCE node
I20260924 03:01:37.355059 22094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:37.355342 22094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:37.356480 22094 hybrid_clock.cc:648] HybridClock initialized: now 1790218897356468 us; error 29 us; skew 500 ppm
Sep 24 03:01:37 dist-test-slave-sb6p krb5kdc[22081](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218897, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:37.358837 22094 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:01:37.359256 22094 webserver.cc:460] Webserver started at http://127.0.98.62:32891/ using document root <none> and password file <none>
I20260924 03:01:37.359442 22094 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:37.359488 22094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:37.359606 22094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:37.360597 22094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "c487a78ab77e4b1cb6ec3ad90e0b9c6d"
format_stamp: "Formatted at 2026-09-24 03:01:37 on dist-test-slave-sb6p"
server_key: "f2f93c1a37372ab5389df705cf0d1efd"
server_key_iv: "c878a9cbbda4db9bc9a307dc3b2cd309"
server_key_version: "encryptionkey@0"
I20260924 03:01:37.360915 22094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "c487a78ab77e4b1cb6ec3ad90e0b9c6d"
format_stamp: "Formatted at 2026-09-24 03:01:37 on dist-test-slave-sb6p"
server_key: "f2f93c1a37372ab5389df705cf0d1efd"
server_key_iv: "c878a9cbbda4db9bc9a307dc3b2cd309"
server_key_version: "encryptionkey@0"
I20260924 03:01:37.362076 22094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:01:37.362728 22109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:37.362880 22094 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:37.362967 22094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "c487a78ab77e4b1cb6ec3ad90e0b9c6d"
format_stamp: "Formatted at 2026-09-24 03:01:37 on dist-test-slave-sb6p"
server_key: "f2f93c1a37372ab5389df705cf0d1efd"
server_key_iv: "c878a9cbbda4db9bc9a307dc3b2cd309"
server_key_version: "encryptionkey@0"
I20260924 03:01:37.363029 22094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:37.366096 22094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:37.368054 22094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:37.368175 22094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:37.371907 22094 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:39125
I20260924 03:01:37.371937 22161 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:39125 every 8 connection(s)
I20260924 03:01:37.372229 22094 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:37.372747 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 22094
I20260924 03:01:37.372851 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:37.373021 392 external_mini_cluster.cc:1428] Setting key d8d316301d1d009f12b7dd2fe52734d7
I20260924 03:01:37.373155 22162 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:37.375254 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:37.373466 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53239 (local address 127.0.98.62:39125)
0924 03:01:37.373629 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:37.373639 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:37.373664 (+ 25us) server_negotiation.cc:407] Connection header received
0924 03:01:37.374147 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:37.374153 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:37.374178 (+ 25us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:37.374286 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:37.374869 (+ 583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:37.374872 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:37.374872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:37.374876 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:37.374909 (+ 33us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:37.374918 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:37.374970 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:37.375044 (+ 74us) server_negotiation.cc:299] Negotiation successful
0924 03:01:37.375074 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":69,"threads_started":1}
I20260924 03:01:37.376084 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:37.376320 22162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d: Bootstrap starting.
I20260924 03:01:37.376981 22162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:37.377300 22162 log.cc:824] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:37.378027 22162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d: No bootstrap required, opened a new log
I20260924 03:01:37.378477 22165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:37.377571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38511 (local address 127.0.98.62:39125)
0924 03:01:37.377606 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:37.377608 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:37.377701 (+ 93us) server_negotiation.cc:407] Connection header received
0924 03:01:37.377775 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:37.377776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:37.377780 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:37.377837 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:37.378297 (+ 460us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:37.378298 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:37.378298 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:37.378302 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:37.378316 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:37.378320 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:37.378378 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:37.378410 (+ 32us) server_negotiation.cc:299] Negotiation successful
0924 03:01:37.378421 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:01:37.380028 22162 raft_consensus.cc:348] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c487a78ab77e4b1cb6ec3ad90e0b9c6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39125 } }
I20260924 03:01:37.380192 22162 raft_consensus.cc:374] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:37.380240 22162 raft_consensus.cc:729] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c487a78ab77e4b1cb6ec3ad90e0b9c6d, State: Initialized, Role: FOLLOWER
I20260924 03:01:37.380357 22162 consensus_queue.cc:260] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c487a78ab77e4b1cb6ec3ad90e0b9c6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39125 } }
I20260924 03:01:37.380441 22162 raft_consensus.cc:388] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:37.380463 22162 raft_consensus.cc:482] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:37.380494 22162 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:37.380975 22162 raft_consensus.cc:504] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c487a78ab77e4b1cb6ec3ad90e0b9c6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39125 } }
I20260924 03:01:37.381055 22162 leader_election.cc:302] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c487a78ab77e4b1cb6ec3ad90e0b9c6d; no voters:
I20260924 03:01:37.381177 22162 leader_election.cc:288] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:37.381242 22172 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:37.381381 22162 sys_catalog.cc:564] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:37.381381 22172 raft_consensus.cc:686] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [term 1 LEADER]: Becoming Leader. State: Replica: c487a78ab77e4b1cb6ec3ad90e0b9c6d, State: Running, Role: LEADER
I20260924 03:01:37.381507 22172 consensus_queue.cc:237] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c487a78ab77e4b1cb6ec3ad90e0b9c6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39125 } }
I20260924 03:01:37.381865 22173 sys_catalog.cc:455] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c487a78ab77e4b1cb6ec3ad90e0b9c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c487a78ab77e4b1cb6ec3ad90e0b9c6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39125 } } }
I20260924 03:01:37.381861 22174 sys_catalog.cc:455] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c487a78ab77e4b1cb6ec3ad90e0b9c6d. Latest consensus state: current_term: 1 leader_uuid: "c487a78ab77e4b1cb6ec3ad90e0b9c6d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c487a78ab77e4b1cb6ec3ad90e0b9c6d" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 39125 } } }
I20260924 03:01:37.381969 22173 sys_catalog.cc:458] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:37.381983 22174 sys_catalog.cc:458] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:37.382166 22181 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:37.382840 22181 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:37.387428 22181 catalog_manager.cc:1347] Generated new cluster ID: d2f712b445c24f528aee5382b4f722e1
I20260924 03:01:37.387476 22181 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:37.407857 22181 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:37.408039 22181 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:37.414829 22181 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c487a78ab77e4b1cb6ec3ad90e0b9c6d: Generated new TSK 0
I20260924 03:01:37.415017 22181 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:37.421730 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 22094
2026-09-24T03:01:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1206 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:37 dist-test-slave-sb6p krb5kdc[22193](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:37 dist-test-slave-sb6p krb5kdc[22193](info): set up 2 sockets
Sep 24 03:01:37 dist-test-slave-sb6p krb5kdc[22193](info): commencing operation
krb5kdc: starting...
W20260924 03:01:38.366307 392 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:38 dist-test-slave-sb6p krb5kdc[22193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:38Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:01:38.473567 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35581
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:44241
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:35581
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:38.551957 22206 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:38.552165 22206 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:38.552196 22206 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:38.553321 22206 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:38.553360 22206 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:38.553378 22206 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:38.553395 22206 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:38.553411 22206 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:38.554880 22206 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:44241
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:35581
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:35581
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22206
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:38.555130 22206 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:38.555336 22206 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:38.559968 22214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:38.559979 22212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:38.559974 22211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:38.560266 22206 server_base.cc:1034] running on GCE node
I20260924 03:01:38.560511 22206 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:38.560792 22206 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:38.561941 22206 hybrid_clock.cc:648] HybridClock initialized: now 1790218898561927 us; error 27 us; skew 500 ppm
Sep 24 03:01:38 dist-test-slave-sb6p krb5kdc[22193](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218898, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:38.570565 22206 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:01:38.570997 22206 webserver.cc:460] Webserver started at http://127.0.98.62:39879/ using document root <none> and password file <none>
I20260924 03:01:38.571203 22206 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:38.571264 22206 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:38.571385 22206 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:38.572412 22206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "b07e47f4dbf2403e8ecaeb2b32cadb91"
format_stamp: "Formatted at 2026-09-24 03:01:38 on dist-test-slave-sb6p"
server_key: "e209f7a5ba3f867d3e508c9cd40845bf"
server_key_iv: "0257c67b82242f7fd11e42fff84e08b9"
server_key_version: "encryptionkey@0"
I20260924 03:01:38.572728 22206 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "b07e47f4dbf2403e8ecaeb2b32cadb91"
format_stamp: "Formatted at 2026-09-24 03:01:38 on dist-test-slave-sb6p"
server_key: "e209f7a5ba3f867d3e508c9cd40845bf"
server_key_iv: "0257c67b82242f7fd11e42fff84e08b9"
server_key_version: "encryptionkey@0"
I20260924 03:01:38.573920 22206 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260924 03:01:38.574589 22221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:38.574721 22206 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:38.574802 22206 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "b07e47f4dbf2403e8ecaeb2b32cadb91"
format_stamp: "Formatted at 2026-09-24 03:01:38 on dist-test-slave-sb6p"
server_key: "e209f7a5ba3f867d3e508c9cd40845bf"
server_key_iv: "0257c67b82242f7fd11e42fff84e08b9"
server_key_version: "encryptionkey@0"
I20260924 03:01:38.574862 22206 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:38.583994 22206 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:38.585705 22206 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:38.585834 22206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:38.589545 22206 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:35581
I20260924 03:01:38.589581 22273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:35581 every 8 connection(s)
I20260924 03:01:38.589886 22206 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:38.590729 22274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:38.592926 22274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91: Bootstrap starting.
I20260924 03:01:38.593434 22274 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:38.593662 22274 log.cc:824] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:38.594337 22274 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91: No bootstrap required, opened a new log
I20260924 03:01:38.595894 22274 raft_consensus.cc:348] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07e47f4dbf2403e8ecaeb2b32cadb91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35581 } }
I20260924 03:01:38.596031 22274 raft_consensus.cc:374] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:38.596052 22274 raft_consensus.cc:729] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b07e47f4dbf2403e8ecaeb2b32cadb91, State: Initialized, Role: FOLLOWER
I20260924 03:01:38.596161 22274 consensus_queue.cc:260] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07e47f4dbf2403e8ecaeb2b32cadb91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35581 } }
I20260924 03:01:38.596240 22274 raft_consensus.cc:388] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:38.596268 22274 raft_consensus.cc:482] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:38.596300 22274 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:38.596763 22274 raft_consensus.cc:504] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07e47f4dbf2403e8ecaeb2b32cadb91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35581 } }
I20260924 03:01:38.596848 22274 leader_election.cc:302] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b07e47f4dbf2403e8ecaeb2b32cadb91; no voters:
I20260924 03:01:38.596973 22274 leader_election.cc:288] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:38.597023 22277 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:38.597203 22274 sys_catalog.cc:564] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:38.597209 22277 raft_consensus.cc:686] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [term 1 LEADER]: Becoming Leader. State: Replica: b07e47f4dbf2403e8ecaeb2b32cadb91, State: Running, Role: LEADER
I20260924 03:01:38.597334 22277 consensus_queue.cc:237] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07e47f4dbf2403e8ecaeb2b32cadb91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35581 } }
I20260924 03:01:38.597672 22279 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b07e47f4dbf2403e8ecaeb2b32cadb91. Latest consensus state: current_term: 1 leader_uuid: "b07e47f4dbf2403e8ecaeb2b32cadb91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07e47f4dbf2403e8ecaeb2b32cadb91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35581 } } }
I20260924 03:01:38.597680 22278 sys_catalog.cc:455] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b07e47f4dbf2403e8ecaeb2b32cadb91" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b07e47f4dbf2403e8ecaeb2b32cadb91" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 35581 } } }
I20260924 03:01:38.597998 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 22206
I20260924 03:01:38.598114 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:38.598289 392 external_mini_cluster.cc:1428] Setting key c823dd8f9015ac57147aa6b6fe226f95
I20260924 03:01:38.597787 22279 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:38.598645 22278 sys_catalog.cc:458] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:38.598691 22289 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:38.599262 22289 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:38.600827 22289 catalog_manager.cc:1347] Generated new cluster ID: 9d2fa65c7bf24dbcb7645a5b73ec7057
I20260924 03:01:38.600878 22289 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 24 03:01:38 dist-test-slave-sb6p krb5kdc[22193](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218898, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:38.608166 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:38.598886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47413 (local address 127.0.98.62:35581)
0924 03:01:38.599063 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:38.599073 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:38.599110 (+ 37us) server_negotiation.cc:407] Connection header received
0924 03:01:38.599286 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:38.599294 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:38.599354 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:38.600197 (+ 843us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:38.600321 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:38.601403 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:38.602125 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:38.602587 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:38.602679 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:38.604808 (+ 2129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:38.604813 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:38.604813 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:38.604825 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:38.606835 (+ 2010us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:38.607130 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:38.607133 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:38.607134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:38.607224 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:38.607391 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:38.607391 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:38.607392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:38.607615 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:38.607775 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:38.607840 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 03:01:38.607974 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":48,"threads_started":1}
I20260924 03:01:38.608968 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:38.613866 22289 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:38.614562 22289 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:38.615782 22295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:38.610160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58481 (local address 127.0.98.62:35581)
0924 03:01:38.610193 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:38.610195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:38.610307 (+ 112us) server_negotiation.cc:407] Connection header received
0924 03:01:38.610347 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:38.610348 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:38.610359 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:38.610398 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:38.610901 (+ 503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:38.611434 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:38.612079 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:38.612297 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:38.612352 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:38.613351 (+ 999us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:38.613352 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:38.613353 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:38.613359 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:38.614830 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:38.615132 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:38.615133 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:38.615133 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:38.615184 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:38.615338 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:38.615339 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:38.615339 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:38.615438 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:38.615527 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:38.615619 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 03:01:38.615697 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":17}
I20260924 03:01:38.623281 22289 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b07e47f4dbf2403e8ecaeb2b32cadb91: Generated new TSK 0
I20260924 03:01:38.623461 22289 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:38.630082 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 22206
2026-09-24T03:01:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1208 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:38 dist-test-slave-sb6p krb5kdc[22305](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:38 dist-test-slave-sb6p krb5kdc[22305](info): set up 2 sockets
Sep 24 03:01:38 dist-test-slave-sb6p krb5kdc[22305](info): commencing operation
krb5kdc: starting...
W20260924 03:01:39.594328 392 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:39 dist-test-slave-sb6p krb5kdc[22305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218899, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:01:39.703132 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34623
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:43871
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:34623
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:39.780221 22318 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:39.780428 22318 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:39.780450 22318 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:39.781589 22318 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:39.781632 22318 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:39.781644 22318 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:39.781664 22318 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:39.781687 22318 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:39.783216 22318 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:43871
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:34623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:34623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22318
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:39.783443 22318 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:39.783645 22318 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:39.788153 22324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:39.788154 22323 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:39.788384 22318 server_base.cc:1034] running on GCE node
W20260924 03:01:39.790596 22326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:39.790886 22318 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:39.791231 22318 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:39.792487 22318 hybrid_clock.cc:648] HybridClock initialized: now 1790218899792469 us; error 32 us; skew 500 ppm
Sep 24 03:01:39 dist-test-slave-sb6p krb5kdc[22305](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:39.795125 22318 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:01:39.795511 22318 webserver.cc:460] Webserver started at http://127.0.98.62:33009/ using document root <none> and password file <none>
I20260924 03:01:39.795697 22318 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:39.795750 22318 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:39.795874 22318 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:39.796962 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "1317118c85704c469a4996dad2956d1b"
format_stamp: "Formatted at 2026-09-24 03:01:39 on dist-test-slave-sb6p"
server_key: "e081b7814c5e07511aeabc247298181e"
server_key_iv: "48cdabd6a610696e11727a3e28c827ad"
server_key_version: "encryptionkey@0"
I20260924 03:01:39.797286 22318 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "1317118c85704c469a4996dad2956d1b"
format_stamp: "Formatted at 2026-09-24 03:01:39 on dist-test-slave-sb6p"
server_key: "e081b7814c5e07511aeabc247298181e"
server_key_iv: "48cdabd6a610696e11727a3e28c827ad"
server_key_version: "encryptionkey@0"
I20260924 03:01:39.798467 22318 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260924 03:01:39.799152 22333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:39.799286 22318 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:01:39.799364 22318 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "1317118c85704c469a4996dad2956d1b"
format_stamp: "Formatted at 2026-09-24 03:01:39 on dist-test-slave-sb6p"
server_key: "e081b7814c5e07511aeabc247298181e"
server_key_iv: "48cdabd6a610696e11727a3e28c827ad"
server_key_version: "encryptionkey@0"
I20260924 03:01:39.799423 22318 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:39.811872 22318 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:39.813834 22318 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:39.813959 22318 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:39.817616 22318 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:34623
I20260924 03:01:39.817649 22385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:34623 every 8 connection(s)
I20260924 03:01:39.817948 22318 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:39.818791 22386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:39.820909 22386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b: Bootstrap starting.
I20260924 03:01:39.821398 22386 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:39.821612 22386 log.cc:824] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:39.822187 22386 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b: No bootstrap required, opened a new log
I20260924 03:01:39.823720 22386 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1317118c85704c469a4996dad2956d1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34623 } }
I20260924 03:01:39.823859 22386 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:39.823879 22386 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1317118c85704c469a4996dad2956d1b, State: Initialized, Role: FOLLOWER
I20260924 03:01:39.823989 22386 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1317118c85704c469a4996dad2956d1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34623 } }
I20260924 03:01:39.824056 22386 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:39.824083 22386 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:39.824115 22386 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:39.824548 22386 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1317118c85704c469a4996dad2956d1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34623 } }
I20260924 03:01:39.824628 22386 leader_election.cc:302] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1317118c85704c469a4996dad2956d1b; no voters:
I20260924 03:01:39.824746 22386 leader_election.cc:288] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:39.824782 22389 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:39.824934 22389 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [term 1 LEADER]: Becoming Leader. State: Replica: 1317118c85704c469a4996dad2956d1b, State: Running, Role: LEADER
I20260924 03:01:39.824955 22386 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:39.825017 22389 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1317118c85704c469a4996dad2956d1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34623 } }
I20260924 03:01:39.825356 22390 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1317118c85704c469a4996dad2956d1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1317118c85704c469a4996dad2956d1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34623 } } }
I20260924 03:01:39.825356 22391 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1317118c85704c469a4996dad2956d1b. Latest consensus state: current_term: 1 leader_uuid: "1317118c85704c469a4996dad2956d1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1317118c85704c469a4996dad2956d1b" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 34623 } } }
I20260924 03:01:39.825452 22390 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:39.825501 22391 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:39.826027 22393 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:39.826567 22393 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:39.827203 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 22318
I20260924 03:01:39.827294 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:39.827452 392 external_mini_cluster.cc:1428] Setting key caab9dab66742d7b30c0960e58b23234
I20260924 03:01:39.828177 22393 catalog_manager.cc:1347] Generated new cluster ID: 134ee0184355446082f29dcfe4d0d960
I20260924 03:01:39.828218 22393 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:39.833652 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:39.827997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51587 (local address 127.0.98.62:34623)
0924 03:01:39.828181 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:39.828190 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:39.828220 (+ 30us) server_negotiation.cc:407] Connection header received
0924 03:01:39.828329 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:39.828336 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:39.828387 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:39.828785 (+ 398us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:39.828876 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:39.830289 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:39.831430 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:39.832054 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:39.832461 (+ 407us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:39.832853 (+ 392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:39.832857 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:39.832857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:39.833052 (+ 195us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:39.833093 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:39.833107 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:39.833339 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:39.833370 (+ 31us) server_negotiation.cc:299] Negotiation successful
0924 03:01:39.833438 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":78,"threads_started":1}
I20260924 03:01:39.834605 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:39.839347 22409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:39.835949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38019 (local address 127.0.98.62:34623)
0924 03:01:39.835989 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:39.835991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:39.836178 (+ 187us) server_negotiation.cc:407] Connection header received
0924 03:01:39.836273 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:39.836274 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:39.836285 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:39.836337 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:39.836503 (+ 166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:39.837170 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:39.838114 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:39.838403 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:39.838439 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:39.839124 (+ 685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:39.839125 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:39.839125 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:39.839131 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:39.839146 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:39.839151 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:39.839204 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:39.839256 (+ 52us) server_negotiation.cc:299] Negotiation successful
0924 03:01:39.839277 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:01:39.846582 22393 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:39.847106 22393 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:39.850890 22393 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1317118c85704c469a4996dad2956d1b: Generated new TSK 0
I20260924 03:01:39.851056 22393 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:39.856015 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 22318
2026-09-24T03:01:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1227 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:39 dist-test-slave-sb6p krb5kdc[22417](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:39 dist-test-slave-sb6p krb5kdc[22417](info): set up 2 sockets
Sep 24 03:01:39 dist-test-slave-sb6p krb5kdc[22417](info): commencing operation
krb5kdc: starting...
W20260924 03:01:40.810369 392 mini_kdc.cc:121] Time spent starting KDC: real 0.936s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:40 dist-test-slave-sb6p krb5kdc[22417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218900, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:01:40.921283 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37049
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:35057
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:37049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:40.999493 22430 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:40.999687 22430 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:40.999708 22430 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:41.000792 22430 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:41.000826 22430 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:41.000838 22430 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:41.000850 22430 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:41.000860 22430 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:41.002251 22430 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:35057
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:37049
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:37049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22430
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:41.002436 22430 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:41.002622 22430 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:41.007149 22436 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:41.007115 22435 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:41.007102 22438 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:41.007313 22430 server_base.cc:1034] running on GCE node
I20260924 03:01:41.007664 22430 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:41.007931 22430 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:41.009068 22430 hybrid_clock.cc:648] HybridClock initialized: now 1790218901009052 us; error 28 us; skew 500 ppm
Sep 24 03:01:41 dist-test-slave-sb6p krb5kdc[22417](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:41.011381 22430 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:01:41.011814 22430 webserver.cc:460] Webserver started at http://127.0.98.62:41515/ using document root <none> and password file <none>
I20260924 03:01:41.012022 22430 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:41.012068 22430 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:41.012189 22430 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:41.013200 22430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "f808f082ad0242b88d58d82b822c693f"
format_stamp: "Formatted at 2026-09-24 03:01:41 on dist-test-slave-sb6p"
server_key: "832517ab2dfaaaa0e8ab575fd6fe8f86"
server_key_iv: "72e34f3c45caaf6de628ddea34949709"
server_key_version: "encryptionkey@0"
I20260924 03:01:41.013526 22430 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "f808f082ad0242b88d58d82b822c693f"
format_stamp: "Formatted at 2026-09-24 03:01:41 on dist-test-slave-sb6p"
server_key: "832517ab2dfaaaa0e8ab575fd6fe8f86"
server_key_iv: "72e34f3c45caaf6de628ddea34949709"
server_key_version: "encryptionkey@0"
I20260924 03:01:41.014719 22430 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:01:41.015421 22445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:41.015568 22430 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:41.015647 22430 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "f808f082ad0242b88d58d82b822c693f"
format_stamp: "Formatted at 2026-09-24 03:01:41 on dist-test-slave-sb6p"
server_key: "832517ab2dfaaaa0e8ab575fd6fe8f86"
server_key_iv: "72e34f3c45caaf6de628ddea34949709"
server_key_version: "encryptionkey@0"
I20260924 03:01:41.015703 22430 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:41.018931 22430 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:41.020558 22430 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:41.020678 22430 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:41.024289 22430 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:37049
I20260924 03:01:41.024326 22497 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:37049 every 8 connection(s)
I20260924 03:01:41.024605 22430 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:41.025382 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 22430
I20260924 03:01:41.025486 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:41.025601 22498 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:41.025632 392 external_mini_cluster.cc:1428] Setting key a90f3d8107d0808ac2817d75fcd4a5ac
I20260924 03:01:41.027876 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:41.026092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38419 (local address 127.0.98.62:37049)
0924 03:01:41.026244 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:41.026253 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:41.026276 (+ 23us) server_negotiation.cc:407] Connection header received
0924 03:01:41.026705 (+ 429us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:41.026711 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:41.026743 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:41.026858 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:41.027477 (+ 619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:41.027480 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:41.027481 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:41.027484 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:41.027518 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:41.027528 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:41.027580 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:41.027659 (+ 79us) server_negotiation.cc:299] Negotiation successful
0924 03:01:41.027691 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":61,"threads_started":1}
I20260924 03:01:41.028800 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:41.028970 22498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f: Bootstrap starting.
I20260924 03:01:41.029532 22498 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:41.029928 22498 log.cc:824] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:41.030627 22498 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f: No bootstrap required, opened a new log
I20260924 03:01:41.031065 22501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:41.030029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59535 (local address 127.0.98.62:37049)
0924 03:01:41.030073 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:41.030075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:41.030227 (+ 152us) server_negotiation.cc:407] Connection header received
0924 03:01:41.030275 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:41.030276 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:41.030280 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:41.030331 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:41.030902 (+ 571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:41.030903 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:41.030903 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0924 03:01:41.030907 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:41.030918 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0924 03:01:41.030921 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:41.030957 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:41.031005 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 03:01:41.031014 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260924 03:01:41.032573 22498 raft_consensus.cc:348] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f808f082ad0242b88d58d82b822c693f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37049 } }
I20260924 03:01:41.032708 22498 raft_consensus.cc:374] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260924 03:01:41.032740 22498 raft_consensus.cc:729] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f808f082ad0242b88d58d82b822c693f, State: Initialized, Role: FOLLOWER
I20260924 03:01:41.032858 22498 consensus_queue.cc:260] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f808f082ad0242b88d58d82b822c693f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37049 } }
I20260924 03:01:41.032928 22498 raft_consensus.cc:388] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260924 03:01:41.032958 22498 raft_consensus.cc:482] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260924 03:01:41.033002 22498 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:41.033551 22498 raft_consensus.cc:504] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f808f082ad0242b88d58d82b822c693f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37049 } }
I20260924 03:01:41.033645 22498 leader_election.cc:302] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f808f082ad0242b88d58d82b822c693f; no voters:
I20260924 03:01:41.033811 22498 leader_election.cc:288] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260924 03:01:41.033877 22508 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:41.034029 22498 sys_catalog.cc:564] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:41.034078 22508 raft_consensus.cc:686] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [term 1 LEADER]: Becoming Leader. State: Replica: f808f082ad0242b88d58d82b822c693f, State: Running, Role: LEADER
I20260924 03:01:41.034196 22508 consensus_queue.cc:237] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f808f082ad0242b88d58d82b822c693f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37049 } }
I20260924 03:01:41.034585 22509 sys_catalog.cc:455] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f808f082ad0242b88d58d82b822c693f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f808f082ad0242b88d58d82b822c693f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37049 } } }
I20260924 03:01:41.034569 22508 sys_catalog.cc:455] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [sys.catalog]: SysCatalogTable state changed. Reason: New leader f808f082ad0242b88d58d82b822c693f. Latest consensus state: current_term: 1 leader_uuid: "f808f082ad0242b88d58d82b822c693f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f808f082ad0242b88d58d82b822c693f" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 37049 } } }
I20260924 03:01:41.034718 22509 sys_catalog.cc:458] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:41.034724 22508 sys_catalog.cc:458] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:41.034909 22517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:41.035455 22517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:41.037147 22517 catalog_manager.cc:1347] Generated new cluster ID: 2cdcf00109cd4adeba9d51255cb8be2c
I20260924 03:01:41.037202 22517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:41.054469 22517 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:41.054672 22517 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:41.061949 22517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f808f082ad0242b88d58d82b822c693f: Generated new TSK 0
I20260924 03:01:41.062134 22517 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260924 03:01:41.064182 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 22430
2026-09-24T03:01:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1206 ms)
[----------] 18 tests from AuthTokenIssuingTest (21777 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:41 dist-test-slave-sb6p krb5kdc[22529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:41 dist-test-slave-sb6p krb5kdc[22529](info): set up 2 sockets
Sep 24 03:01:41 dist-test-slave-sb6p krb5kdc[22529](info): commencing operation
krb5kdc: starting...
W20260924 03:01:42.010339 392 mini_kdc.cc:121] Time spent starting KDC: real 0.929s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:42 dist-test-slave-sb6p krb5kdc[22529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:01:42.117086 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45395
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45395,127.0.98.61:36413,127.0.98.60:32959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:42.194346 22542 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:01:42.194578 22542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:42.194597 22542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:42.194615 22542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:42.195767 22542 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:42.195807 22542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:42.195820 22542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:42.195832 22542 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:42.195842 22542 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:42.197296 22542 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45395,127.0.98.61:36413,127.0.98.60:32959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:45395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:42.197520 22542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:42.197721 22542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:42.202184 22547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:42.202219 22548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:42.202373 22542 server_base.cc:1034] running on GCE node
W20260924 03:01:42.202219 22550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:42.202652 22542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:42.202920 22542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:42.204058 22542 hybrid_clock.cc:648] HybridClock initialized: now 1790218902204037 us; error 37 us; skew 500 ppm
Sep 24 03:01:42 dist-test-slave-sb6p krb5kdc[22529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:42.206357 22542 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:01:42.206828 22542 webserver.cc:460] Webserver started at http://127.0.98.62:43719/ using document root <none> and password file <none>
I20260924 03:01:42.207021 22542 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:42.207068 22542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:42.207188 22542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:42.208194 22542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "124f912fb6db485e871ba041d41aa647"
format_stamp: "Formatted at 2026-09-24 03:01:42 on dist-test-slave-sb6p"
server_key: "da5ba9b61cada6047ee67b5e797ada32"
server_key_iv: "35dd2b4aff5db671c79496c7b0109246"
server_key_version: "encryptionkey@0"
I20260924 03:01:42.208518 22542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "124f912fb6db485e871ba041d41aa647"
format_stamp: "Formatted at 2026-09-24 03:01:42 on dist-test-slave-sb6p"
server_key: "da5ba9b61cada6047ee67b5e797ada32"
server_key_iv: "35dd2b4aff5db671c79496c7b0109246"
server_key_version: "encryptionkey@0"
I20260924 03:01:42.209746 22542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260924 03:01:42.210583 22557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:42.210825 22542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:01:42.210913 22542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "124f912fb6db485e871ba041d41aa647"
format_stamp: "Formatted at 2026-09-24 03:01:42 on dist-test-slave-sb6p"
server_key: "da5ba9b61cada6047ee67b5e797ada32"
server_key_iv: "35dd2b4aff5db671c79496c7b0109246"
server_key_version: "encryptionkey@0"
I20260924 03:01:42.210983 22542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:42.228387 22542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:42.230263 22542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:42.230402 22542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:42.234277 22542 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:45395
I20260924 03:01:42.234315 22609 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:45395 every 8 connection(s)
I20260924 03:01:42.234675 22542 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:42.235634 22610 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:42.237133 22610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:01:42.241061 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 22542
I20260924 03:01:42.241159 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:42.241314 392 external_mini_cluster.cc:1428] Setting key f071839c36878c2e54cc51745350f018
Sep 24 03:01:42 dist-test-slave-sb6p krb5kdc[22529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:42.246068 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.237729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45395 (local address 127.0.0.1:43840)
0924 03:01:42.237886 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.237924 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:42.238371 (+ 447us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:42.238630 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:42.238632 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:42.238900 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:42.239055 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.239059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.240001 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.240001 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.240654 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.240656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.241148 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.241150 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.241212 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.241724 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:42.241732 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:42.243186 (+ 1454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:42.244776 (+ 1590us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.244782 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.244784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.244960 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.245158 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.245159 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.245159 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.245237 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.245547 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:42.245551 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:42.245611 (+ 60us) client_negotiation.cc:769] Sending connection context
0924 03:01:42.245693 (+ 82us) client_negotiation.cc:240] Negotiation successful
0924 03:01:42.245902 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":76,"thread_start_us":60,"threads_started":1}
I20260924 03:01:42.246068 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.237788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43840 (local address 127.0.98.62:45395)
0924 03:01:42.237922 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.237930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.237998 (+ 68us) server_negotiation.cc:407] Connection header received
0924 03:01:42.238489 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.238490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.238506 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.238614 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.239125 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.239930 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.240713 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.241050 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.241147 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.243266 (+ 2119us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:42.243272 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:42.243273 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:42.243279 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:42.244705 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.245026 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.245028 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.245029 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.245101 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.245288 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.245288 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.245289 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.245489 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:42.245559 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:42.245723 (+ 164us) server_negotiation.cc:299] Negotiation successful
0924 03:01:42.245833 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":67,"threads_started":1}
I20260924 03:01:42.246789 22610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:01:42.247116 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.247012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36413 (local address 127.0.0.1:41020)
0924 03:01:42.247052 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.247092 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 03:01:42.247233 22561 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36413: connect: Connection refused (error 111)
I20260924 03:01:42.247604 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.247521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36413 (local address 127.0.0.1:41022)
0924 03:01:42.247558 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.247581 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260924 03:01:42.247747 22610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36413: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36413: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 03:01:42.264915 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36413
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45395,127.0.98.61:36413,127.0.98.60:32959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:01:42.296960 22610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } attempt: 1
I20260924 03:01:42.297430 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.297311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36413 (local address 127.0.0.1:41034)
0924 03:01:42.297366 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.297400 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 03:01:42.297776 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.297692 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36413 (local address 127.0.0.1:41042)
0924 03:01:42.297733 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.297757 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 03:01:42.297879 22610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36413: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36413: connect: Connection refused (error 111)
W20260924 03:01:42.342787 22617 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:01:42.342994 22617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:42.343010 22617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:42.343029 22617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:42.344194 22617 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:42.344237 22617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:42.344249 22617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:42.344260 22617 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:42.344275 22617 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:42.345741 22617 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45395,127.0.98.61:36413,127.0.98.60:32959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:36413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22617
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:42.345978 22617 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:42.346191 22617 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:42.350723 22625 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:42.350880 22617 server_base.cc:1034] running on GCE node
W20260924 03:01:42.350733 22623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:42.350766 22622 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:42.351181 22617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:42.351418 22617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:42.352550 22617 hybrid_clock.cc:648] HybridClock initialized: now 1790218902352536 us; error 26 us; skew 500 ppm
Sep 24 03:01:42 dist-test-slave-sb6p krb5kdc[22529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:42.354938 22617 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 03:01:42.355301 22617 webserver.cc:460] Webserver started at http://127.0.98.61:41427/ using document root <none> and password file <none>
I20260924 03:01:42.355465 22617 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:42.355502 22617 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:42.355589 22617 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:42.356571 22617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-1/data/instance:
uuid: "18cc6a9ad9874d439cfbc151fcef2260"
format_stamp: "Formatted at 2026-09-24 03:01:42 on dist-test-slave-sb6p"
server_key: "c7da52c71083f308bcf045fd4846d847"
server_key_iv: "1b294259a8995445faca1b1f63be39c6"
server_key_version: "encryptionkey@0"
I20260924 03:01:42.356860 22617 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-1/wal/instance:
uuid: "18cc6a9ad9874d439cfbc151fcef2260"
format_stamp: "Formatted at 2026-09-24 03:01:42 on dist-test-slave-sb6p"
server_key: "c7da52c71083f308bcf045fd4846d847"
server_key_iv: "1b294259a8995445faca1b1f63be39c6"
server_key_version: "encryptionkey@0"
I20260924 03:01:42.357973 22617 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:01:42.358592 22632 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:42.358745 22617 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:42.358819 22617 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-1/wal
uuid: "18cc6a9ad9874d439cfbc151fcef2260"
format_stamp: "Formatted at 2026-09-24 03:01:42 on dist-test-slave-sb6p"
server_key: "c7da52c71083f308bcf045fd4846d847"
server_key_iv: "1b294259a8995445faca1b1f63be39c6"
server_key_version: "encryptionkey@0"
I20260924 03:01:42.358872 22617 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:42.366014 22610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } attempt: 2
I20260924 03:01:42.366369 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.366262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36413 (local address 127.0.0.1:41050)
0924 03:01:42.366311 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.366341 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 03:01:42.366703 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.366622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36413 (local address 127.0.0.1:41056)
0924 03:01:42.366656 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.366679 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36413: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260924 03:01:42.366801 22610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:36413: Network error: Client connection negotiation failed: client connection to 127.0.98.61:36413: connect: Connection refused (error 111)
I20260924 03:01:42.376487 22617 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:42.378118 22617 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:42.378234 22617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:42.381886 22617 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:36413
I20260924 03:01:42.381915 22684 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:36413 every 8 connection(s)
I20260924 03:01:42.382217 22617 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-1/data/info.pb
I20260924 03:01:42.383112 22685 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:42.384541 22685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:01:42.389191 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 22617
I20260924 03:01:42.389285 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-1/wal/instance
I20260924 03:01:42.389444 392 external_mini_cluster.cc:1428] Setting key edf078ed3aa9d92296da6fd7626cf26d
Sep 24 03:01:42 dist-test-slave-sb6p krb5kdc[22529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:42.392586 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.385140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43846 (local address 127.0.98.62:45395)
0924 03:01:42.385208 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.385210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.385365 (+ 155us) server_negotiation.cc:407] Connection header received
0924 03:01:42.385804 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.385805 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.385814 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.385861 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.386382 (+ 521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.386881 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.387770 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.387947 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.387988 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.390006 (+ 2018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:42.390008 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:42.390008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:42.390014 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:42.391418 (+ 1404us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.391768 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.391769 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.391770 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.391829 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.392029 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.392030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.392030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.392154 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:42.392213 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:42.392428 (+ 215us) server_negotiation.cc:299] Negotiation successful
0924 03:01:42.392501 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260924 03:01:42.392669 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.385101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45395 (local address 127.0.0.1:43846)
0924 03:01:42.385252 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.385293 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:42.385723 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:42.385913 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:42.385916 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:42.386136 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:42.386320 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.386324 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.386950 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.386950 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.387707 (+ 757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.387708 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.388005 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.388006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.388104 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.388478 (+ 374us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:42.388486 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:42.389918 (+ 1432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:42.391500 (+ 1582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.391506 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.391507 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.391704 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.391888 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.391888 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.391889 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.391979 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.392205 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:42.392208 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:42.392324 (+ 116us) client_negotiation.cc:769] Sending connection context
0924 03:01:42.392398 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 03:01:42.392509 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
I20260924 03:01:42.393181 22685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
Sep 24 03:01:42 dist-test-slave-sb6p krb5kdc[22529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:01:42.402698 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.393416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36413 (local address 127.0.0.1:41070)
0924 03:01:42.393454 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.393465 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:42.393499 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:42.393766 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:42.393767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:42.393868 (+ 101us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:42.393913 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.393914 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.395141 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.395142 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.395624 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.395626 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.396222 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.396222 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.396283 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.396642 (+ 359us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:42.396646 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:42.400110 (+ 3464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:42.401637 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.401659 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.401659 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.401845 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.402043 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.402044 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.402044 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.402103 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.402344 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:42.402344 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:42.402391 (+ 47us) client_negotiation.cc:769] Sending connection context
0924 03:01:42.402465 (+ 74us) client_negotiation.cc:240] Negotiation successful
0924 03:01:42.402596 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260924 03:01:42.402794 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.393473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41070 (local address 127.0.98.61:36413)
0924 03:01:42.393618 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.393622 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.393643 (+ 21us) server_negotiation.cc:407] Connection header received
0924 03:01:42.393664 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.393666 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.393679 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.393757 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.393974 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.395075 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.395697 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.396155 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.396227 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.400185 (+ 3958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:42.400194 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:42.400194 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:42.400201 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:42.401562 (+ 1361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.401929 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.401931 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.401931 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.401982 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.402163 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.402163 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.402163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.402292 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:42.402346 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:42.402488 (+ 142us) server_negotiation.cc:299] Negotiation successful
0924 03:01:42.402682 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":94,"server-negotiator.queue_time_us":117,"thread_start_us":81,"threads_started":1}
I20260924 03:01:42.403331 22685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:01:42.403607 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.403502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32959 (local address 127.0.0.1:37256)
0924 03:01:42.403534 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.403584 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260924 03:01:42.403733 22634 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32959: connect: Connection refused (error 111)
I20260924 03:01:42.404075 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.404001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32959 (local address 127.0.0.1:37272)
0924 03:01:42.404027 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.404054 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 03:01:42.404166 22685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32959: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32959: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 03:01:42.413242 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32959
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:34439
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:45395,127.0.98.61:36413,127.0.98.60:32959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:01:42.453365 22685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } attempt: 1
I20260924 03:01:42.453974 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.453690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32959 (local address 127.0.0.1:37274)
0924 03:01:42.453880 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.453933 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260924 03:01:42.454353 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.454246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32959 (local address 127.0.0.1:37286)
0924 03:01:42.454288 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.454327 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 03:01:42.454445 22685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32959: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32959: connect: Connection refused (error 111)
I20260924 03:01:42.457937 22610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } attempt: 3
Sep 24 03:01:42 dist-test-slave-sb6p krb5kdc[22529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:01:42.464795 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.458206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36413 (local address 127.0.0.1:41080)
0924 03:01:42.458251 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.458263 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:42.458311 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:42.458471 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:42.458472 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:42.458635 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:42.458696 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.458698 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.459588 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.459589 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.460116 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.460117 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.460569 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.460570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.460629 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.461068 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:42.461072 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:42.462367 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:42.463979 (+ 1612us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.463980 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.463980 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.464127 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.464308 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.464309 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.464309 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.464352 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.464564 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:42.464564 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:42.464600 (+ 36us) client_negotiation.cc:769] Sending connection context
0924 03:01:42.464643 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 03:01:42.464711 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260924 03:01:42.464830 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.458289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41080 (local address 127.0.98.61:36413)
0924 03:01:42.458342 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.458345 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.458355 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:01:42.458381 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.458382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.458393 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.458466 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.458758 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.459516 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.460180 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.460501 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.460560 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.462449 (+ 1889us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:42.462451 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:42.462451 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:42.462459 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:42.463893 (+ 1434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.464186 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.464187 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.464188 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.464250 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.464389 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.464390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.464390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.464515 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:42.464576 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:42.464668 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 03:01:42.464742 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 03:01:42.465374 22610 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:01:42.465669 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.465584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32959 (local address 127.0.0.1:37296)
0924 03:01:42.465622 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.465647 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260924 03:01:42.465952 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.465897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32959 (local address 127.0.0.1:37298)
0924 03:01:42.465921 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.465940 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260924 03:01:42.466034 22610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32959: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32959: connect: Connection refused (error 111)
W20260924 03:01:42.494956 22692 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:01:42.495163 22692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:42.495179 22692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:42.495198 22692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:42.496269 22692 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:42.496304 22692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:42.496315 22692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:42.496326 22692 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:42.496340 22692 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:42.497210 22610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } attempt: 1
I20260924 03:01:42.497637 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.497504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32959 (local address 127.0.0.1:37300)
0924 03:01:42.497561 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.497606 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 03:01:42.497957 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.497856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32959 (local address 127.0.0.1:37308)
0924 03:01:42.497904 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.497933 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 03:01:42.497936 22692 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:34439
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:45395,127.0.98.61:36413,127.0.98.60:32959
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:32959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
W20260924 03:01:42.498059 22610 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32959: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32959: connect: Connection refused (error 111)
I20260924 03:01:42.498178 22692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:42.498386 22692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:42.502920 22697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:42.502938 22698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:42.502920 22700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:42.503129 22692 server_base.cc:1034] running on GCE node
I20260924 03:01:42.503396 22692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:42.503655 22692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:42.504794 22692 hybrid_clock.cc:648] HybridClock initialized: now 1790218902504770 us; error 36 us; skew 500 ppm
Sep 24 03:01:42 dist-test-slave-sb6p krb5kdc[22529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:42.507053 22692 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 03:01:42.507462 22692 webserver.cc:460] Webserver started at http://127.0.98.60:37855/ using document root <none> and password file <none>
I20260924 03:01:42.507642 22692 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:42.507679 22692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:42.507777 22692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:42.508755 22692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-2/data/instance:
uuid: "26aa13c97dc641249b3c47ce3829e2e9"
format_stamp: "Formatted at 2026-09-24 03:01:42 on dist-test-slave-sb6p"
server_key: "6583705b775ba685fe14505245e3071a"
server_key_iv: "873d94b1b13eaef985e6a75fda804777"
server_key_version: "encryptionkey@0"
I20260924 03:01:42.509035 22692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-2/wal/instance:
uuid: "26aa13c97dc641249b3c47ce3829e2e9"
format_stamp: "Formatted at 2026-09-24 03:01:42 on dist-test-slave-sb6p"
server_key: "6583705b775ba685fe14505245e3071a"
server_key_iv: "873d94b1b13eaef985e6a75fda804777"
server_key_version: "encryptionkey@0"
I20260924 03:01:42.510144 22692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:01:42.510789 22707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:42.510922 22692 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:01:42.511003 22692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-2/wal
uuid: "26aa13c97dc641249b3c47ce3829e2e9"
format_stamp: "Formatted at 2026-09-24 03:01:42 on dist-test-slave-sb6p"
server_key: "6583705b775ba685fe14505245e3071a"
server_key_iv: "873d94b1b13eaef985e6a75fda804777"
server_key_version: "encryptionkey@0"
I20260924 03:01:42.511062 22692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:42.521667 22692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:42.522593 22685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } attempt: 2
I20260924 03:01:42.522958 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.522826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32959 (local address 127.0.0.1:37324)
0924 03:01:42.522885 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.522923 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 03:01:42.523262 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.523179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32959 (local address 127.0.0.1:37326)
0924 03:01:42.523216 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.523239 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32959: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 03:01:42.523366 22685 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:32959: Network error: Client connection negotiation failed: client connection to 127.0.98.60:32959: connect: Connection refused (error 111)
I20260924 03:01:42.523590 22692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:42.523694 22692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:42.527230 22692 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:32959
I20260924 03:01:42.527240 22759 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:32959 every 8 connection(s)
I20260924 03:01:42.527573 22692 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-2/data/info.pb
I20260924 03:01:42.528474 22760 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:42.529986 22760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:01:42 dist-test-slave-sb6p krb5kdc[22529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:42.537477 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 22692
I20260924 03:01:42.537563 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790218647334776-392-0/minicluster-data/master-2/wal/instance
I20260924 03:01:42.537738 392 external_mini_cluster.cc:1428] Setting key 4fa95a715d718cafd43e7a786fc92d30
I20260924 03:01:42.537855 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.530650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43858 (local address 127.0.98.62:45395)
0924 03:01:42.530706 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.530708 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.530901 (+ 193us) server_negotiation.cc:407] Connection header received
0924 03:01:42.531350 (+ 449us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.531351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.531361 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.531407 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.531949 (+ 542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.532403 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.533398 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.533584 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.533635 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.535532 (+ 1897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:42.535533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:42.535533 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:42.535540 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:42.536738 (+ 1198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.537044 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.537045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.537045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.537088 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.537247 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.537248 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.537248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.537342 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:42.537396 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:42.537683 (+ 287us) server_negotiation.cc:299] Negotiation successful
0924 03:01:42.537761 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260924 03:01:42.538077 22762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.530639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:45395 (local address 127.0.0.1:43858)
0924 03:01:42.530786 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.530821 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:42.531262 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:42.531459 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:42.531462 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:42.531695 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:42.531885 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.531889 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.532462 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.532463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.533327 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.533330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.533652 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.533653 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.533760 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.534174 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:42.534182 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:42.535450 (+ 1268us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:42.536804 (+ 1354us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.536807 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.536808 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.536996 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.537128 (+ 132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.537128 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.537128 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.537209 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.537408 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:42.537416 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:42.537548 (+ 132us) client_negotiation.cc:769] Sending connection context
0924 03:01:42.537651 (+ 103us) client_negotiation.cc:240] Negotiation successful
0924 03:01:42.537793 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
I20260924 03:01:42.538645 22760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:01:42 dist-test-slave-sb6p krb5kdc[22529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
Sep 24 03:01:42 dist-test-slave-sb6p krb5kdc[22529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:01:42.544818 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.538915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41084 (local address 127.0.98.61:36413)
0924 03:01:42.538955 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.538956 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.538997 (+ 41us) server_negotiation.cc:407] Connection header received
0924 03:01:42.539064 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.539065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.539073 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.539116 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.539391 (+ 275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.539850 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.540508 (+ 658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.540700 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.540761 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.542444 (+ 1683us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:42.542445 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:42.542445 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:42.542451 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:42.543809 (+ 1358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.544134 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.544135 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.544136 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.544190 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.544355 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.544355 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.544356 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.544484 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:42.544553 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:42.544684 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 03:01:42.544750 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:01:42.544814 22762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.538860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:36413 (local address 127.0.0.1:41084)
0924 03:01:42.538954 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.538964 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:42.539009 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:42.539117 (+ 108us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:42.539118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:42.539262 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:42.539315 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.539316 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.539913 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.539914 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.540433 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.540434 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.540755 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.540756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.540814 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.541153 (+ 339us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:42.541158 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:42.542368 (+ 1210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:42.543885 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.543886 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.543886 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.544072 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.544250 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.544250 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.544250 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.544306 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.544542 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:42.544542 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:42.544576 (+ 34us) client_negotiation.cc:769] Sending connection context
0924 03:01:42.544636 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 03:01:42.544721 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":81,"mutex_wait_us":8}
I20260924 03:01:42.545214 22760 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:01:42.545861 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.538104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37342 (local address 127.0.98.60:32959)
0924 03:01:42.538257 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.538261 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.538399 (+ 138us) server_negotiation.cc:407] Connection header received
0924 03:01:42.538439 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.538442 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.538457 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.538555 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.539174 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.539929 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.540589 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.540905 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.540971 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.543081 (+ 2110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:42.543088 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:42.543089 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:42.543095 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:42.544618 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.544985 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.544989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.544989 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.545048 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.545292 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.545292 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.545293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.545461 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:42.545605 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:42.545656 (+ 51us) server_negotiation.cc:299] Negotiation successful
0924 03:01:42.545756 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":131,"thread_start_us":72,"threads_started":1}
Sep 24 03:01:42 dist-test-slave-sb6p krb5kdc[22529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 03:01:42 dist-test-slave-sb6p krb5kdc[22529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:42.552278 22762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.545416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32959 (local address 127.0.0.1:37348)
0924 03:01:42.545449 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.545460 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:42.545497 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:42.545694 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:42.545695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:42.545835 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:42.545868 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.545870 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.546614 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.546615 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.547323 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.547326 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.547691 (+ 365us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.547692 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.547752 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.548079 (+ 327us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:42.548084 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:42.549343 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:42.551379 (+ 2036us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.551380 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.551380 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.551523 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.551703 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.551704 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.551704 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.551744 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.551966 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:42.551967 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:42.552009 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 03:01:42.552073 (+ 64us) client_negotiation.cc:240] Negotiation successful
0924 03:01:42.552147 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260924 03:01:42.552441 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.545456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37348 (local address 127.0.98.60:32959)
0924 03:01:42.545591 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.545592 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.545598 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:01:42.545612 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.545613 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.545621 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.545929 (+ 308us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.545951 (+ 22us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.546538 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.547395 (+ 857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.547624 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.547669 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.549410 (+ 1741us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:42.549411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:42.549412 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:42.549417 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:42.551312 (+ 1895us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.551587 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.551588 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.551589 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.551644 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.551792 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.551793 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.551793 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.551911 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:42.551955 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:42.552100 (+ 145us) server_negotiation.cc:299] Negotiation successful
0924 03:01:42.552205 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":27,"server-negotiator.queue_time_us":119,"thread_start_us":91,"threads_started":1}
I20260924 03:01:42.552897 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.546603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43862 (local address 127.0.98.62:45395)
0924 03:01:42.546639 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.546640 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.546650 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:01:42.546673 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.546674 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.546683 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.546725 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.547212 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.547641 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.548262 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.548542 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.548611 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.550411 (+ 1800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:42.550413 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:42.550413 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:42.550421 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:42.551771 (+ 1350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.552064 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.552065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.552066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.552119 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.552291 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.552291 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.552291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.552411 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:42.552573 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:42.552748 (+ 175us) server_negotiation.cc:299] Negotiation successful
0924 03:01:42.552811 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260924 03:01:42.553910 22760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9: Bootstrap starting.
I20260924 03:01:42.554608 22760 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:42.554932 22760 log.cc:824] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:42.555652 22760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9: No bootstrap required, opened a new log
I20260924 03:01:42.557075 22760 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } }
I20260924 03:01:42.557233 22760 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26aa13c97dc641249b3c47ce3829e2e9, State: Initialized, Role: FOLLOWER
I20260924 03:01:42.557317 22760 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } }
Sep 24 03:01:42 dist-test-slave-sb6p krb5kdc[22529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218902, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:01:42.557627 22760 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:42.557574 22767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } } }
I20260924 03:01:42.557687 22767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9 [sys.catalog]: This master's current role is: FOLLOWER
W20260924 03:01:42.559549 22778 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:01:42.559643 22778 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:01:42.560524 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.553433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41094 (local address 127.0.98.61:36413)
0924 03:01:42.553470 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.553472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.553498 (+ 26us) server_negotiation.cc:407] Connection header received
0924 03:01:42.553517 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.553518 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.553528 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.553574 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.553958 (+ 384us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.554645 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.555455 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.555688 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.555753 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.557954 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:42.557955 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:42.557956 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:42.557963 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:42.559457 (+ 1494us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.559786 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.559787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.559787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.559844 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.560025 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.560025 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.560026 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.560147 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:42.560311 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:42.560348 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 03:01:42.560418 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:01:42.559672 22692 master_runner.cc:186] Error getting master registration for 127.0.98.62:45395: OK, instance_id { permanent_uuid: "124f912fb6db485e871ba041d41aa647" instance_seqno: 1790218902233232 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:01:42.561805 22692 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:01:42.561942 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260924 03:01:42.562956 22584 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "124f912fb6db485e871ba041d41aa647"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43862
I20260924 03:01:42.573215 22610 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } attempt: 2
Sep 24 03:01:42 dist-test-slave-sb6p krb5kdc[22529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:01:42.580524 22612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.573498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32959 (local address 127.0.0.1:37358)
0924 03:01:42.573537 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.573562 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:42.573617 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:42.573759 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:42.573759 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:42.573941 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:42.573993 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.573995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.574947 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.574947 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.575751 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.575754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.576237 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.576237 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.576289 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.576741 (+ 452us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:42.576746 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:42.577982 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:42.579577 (+ 1595us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.579578 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.579579 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.579744 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.579906 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.579907 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.579907 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.579957 (+ 50us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.580223 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:42.580224 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:42.580270 (+ 46us) client_negotiation.cc:769] Sending connection context
0924 03:01:42.580331 (+ 61us) client_negotiation.cc:240] Negotiation successful
0924 03:01:42.580425 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260924 03:01:42.580538 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.573564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37358 (local address 127.0.98.60:32959)
0924 03:01:42.573604 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.573606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.573615 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:01:42.573671 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.573672 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.573681 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.573750 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.574060 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.574876 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.575827 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.576163 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.576232 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.578062 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:42.578064 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:42.578064 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:42.578071 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:42.579499 (+ 1428us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.579802 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.579803 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.579804 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.579855 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.580003 (+ 148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.580004 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.580004 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.580161 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:42.580220 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:42.580364 (+ 144us) server_negotiation.cc:299] Negotiation successful
0924 03:01:42.580435 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260924 03:01:42.583509 22610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647: Bootstrap starting.
I20260924 03:01:42.584326 22610 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:42.584626 22610 log.cc:824] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:42.587137 22610 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647: No bootstrap required, opened a new log
I20260924 03:01:42.588513 22610 raft_consensus.cc:348] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } }
I20260924 03:01:42.588680 22610 raft_consensus.cc:729] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 124f912fb6db485e871ba041d41aa647, State: Initialized, Role: FOLLOWER
I20260924 03:01:42.588768 22610 consensus_queue.cc:260] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } }
I20260924 03:01:42.589067 22610 sys_catalog.cc:564] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:42.589028 22781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } } }
I20260924 03:01:42.589121 22781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:01:42.589166 22584 raft_consensus.cc:482] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 03:01:42.589258 22584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:42.589949 22584 raft_consensus.cc:504] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } }
I20260924 03:01:42.590718 22734 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "124f912fb6db485e871ba041d41aa647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "26aa13c97dc641249b3c47ce3829e2e9"
I20260924 03:01:42.590873 22734 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:42.591739 22734 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124f912fb6db485e871ba041d41aa647 in term 1.
I20260924 03:01:42.592149 22559 leader_election.cc:302] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 124f912fb6db485e871ba041d41aa647, 26aa13c97dc641249b3c47ce3829e2e9; no voters:
I20260924 03:01:42.592268 22781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:42.592396 22781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [term 1 LEADER]: Becoming Leader. State: Replica: 124f912fb6db485e871ba041d41aa647, State: Running, Role: LEADER
I20260924 03:01:42.592480 22781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } }
I20260924 03:01:42.592535 22584 leader_election.cc:288] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [CANDIDATE]: Term 1 election: Requested vote from peers 18cc6a9ad9874d439cfbc151fcef2260 (127.0.98.61:36413), 26aa13c97dc641249b3c47ce3829e2e9 (127.0.98.60:32959)
I20260924 03:01:42.592880 22781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 124f912fb6db485e871ba041d41aa647. Latest consensus state: current_term: 1 leader_uuid: "124f912fb6db485e871ba041d41aa647" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } } }
I20260924 03:01:42.592947 22781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [sys.catalog]: This master's current role is: LEADER
W20260924 03:01:42.594167 22792 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:01:42.594254 22792 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:01:42.594111 22542 master_runner.cc:186] Error getting master registration for 127.0.98.61:36413: OK, instance_id { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" instance_seqno: 1790218902380899 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:01:42.596125 22542 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:01:42.596261 22796 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:42.596473 22796 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:42.597874 22734 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9 [term 1 FOLLOWER]: Refusing update from remote peer 124f912fb6db485e871ba041d41aa647: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:01:42.598253 22794 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:01:42.599534 22767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 124f912fb6db485e871ba041d41aa647. Latest consensus state: current_term: 1 leader_uuid: "124f912fb6db485e871ba041d41aa647" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } } }
I20260924 03:01:42.599622 22767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:01:42.600538 22794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "124f912fb6db485e871ba041d41aa647" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } } }
I20260924 03:01:42.600642 22794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:42.600612 22767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "124f912fb6db485e871ba041d41aa647" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } } }
I20260924 03:01:42.600675 22767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:01:42.600926 22804 mvcc.cc:205] Tried to move back new op lower bound from 7332736625038688256 to 7332736625019383808. Current Snapshot: MvccSnapshot[applied={T|T < 7332736625038688256}]
I20260924 03:01:42.601163 22805 mvcc.cc:205] Tried to move back new op lower bound from 7332736625038688256 to 7332736625019383808. Current Snapshot: MvccSnapshot[applied={T|T < 7332736625038688256}]
I20260924 03:01:42.607999 22796 catalog_manager.cc:1347] Generated new cluster ID: 3917fee02809401dabe9004f9267e32e
I20260924 03:01:42.608050 22796 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:42.614503 22685 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } attempt: 3
I20260924 03:01:42.617496 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.594774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41102 (local address 127.0.98.61:36413)
0924 03:01:42.594813 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.594815 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.602172 (+ 7357us) server_negotiation.cc:407] Connection header received
0924 03:01:42.602241 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.602242 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.602254 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.602309 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.603190 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.603951 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.605278 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.606029 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.606097 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.612637 (+ 6540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:42.612639 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:42.612639 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:42.612647 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:42.614331 (+ 1684us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.614856 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.614857 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.614857 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.614916 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.615103 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.615104 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.615104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.615237 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:42.615327 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:42.617332 (+ 2005us) server_negotiation.cc:299] Negotiation successful
0924 03:01:42.617410 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:01:42.618466 22796 catalog_manager.cc:1370] Generated new certificate authority record
Sep 24 03:01:42 dist-test-slave-sb6p krb5kdc[22529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:01:42.619714 22796 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:42.623745 22687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.614763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:32959 (local address 127.0.0.1:37380)
0924 03:01:42.614845 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:42.614854 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:42.614888 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:42.615095 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:42.615096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:42.615245 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:42.615295 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.615297 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.616172 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.616173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.616945 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:42.616947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.617447 (+ 500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.617448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:42.617502 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.617870 (+ 368us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:42.617875 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:42.619303 (+ 1428us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:42.622504 (+ 3201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.622505 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.622505 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.622730 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.622920 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:42.622920 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:42.622921 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:42.622974 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:42.623454 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:42.623455 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:42.623497 (+ 42us) client_negotiation.cc:769] Sending connection context
0924 03:01:42.623557 (+ 60us) client_negotiation.cc:240] Negotiation successful
0924 03:01:42.623646 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260924 03:01:42.623770 22765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.595175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37372 (local address 127.0.98.60:32959)
0924 03:01:42.596200 (+ 1025us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.596203 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.607504 (+ 11301us) server_negotiation.cc:407] Connection header received
0924 03:01:42.608164 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.608177 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.608188 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.608217 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.611036 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.611709 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.615493 (+ 3784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.615783 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.615821 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.620626 (+ 4805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:42.620627 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:42.620628 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:42.620634 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:42.622210 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.622749 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.622750 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.622750 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.622796 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.623027 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.623028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.623028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.623327 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:42.623425 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:42.623616 (+ 191us) server_negotiation.cc:299] Negotiation successful
0924 03:01:42.623683 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1004}
I20260924 03:01:42.624065 22764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.614951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37380 (local address 127.0.98.60:32959)
0924 03:01:42.614987 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.614989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.614995 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:01:42.615012 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.615012 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.615020 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.615060 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.615372 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.616078 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.617032 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.617346 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.617399 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.619392 (+ 1993us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:42.619393 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:42.619393 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:42.619400 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:42.622434 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.622803 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.622805 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.622805 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.622858 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.623030 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.623031 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.623031 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.623155 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:42.623208 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:42.623909 (+ 701us) server_negotiation.cc:299] Negotiation successful
0924 03:01:42.623979 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:01:42.625543 22685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260: Bootstrap starting.
I20260924 03:01:42.626236 22685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:42.626513 22685 log.cc:824] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:42.627400 22685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260: No bootstrap required, opened a new log
I20260924 03:01:42.629091 22613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:42.594548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43876 (local address 127.0.98.62:45395)
0924 03:01:42.594585 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.594587 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.607016 (+ 12429us) server_negotiation.cc:407] Connection header received
0924 03:01:42.607533 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.607534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.607547 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.607604 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.609377 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.610074 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.615894 (+ 5820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:42.616185 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:42.616221 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:42.620353 (+ 4132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:42.620354 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:42.620354 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:42.620361 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:42.622306 (+ 1945us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.628212 (+ 5906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.628214 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.628214 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.628289 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:42.628486 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:42.628487 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:42.628488 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:42.628633 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:42.628709 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:42.628903 (+ 194us) server_negotiation.cc:299] Negotiation successful
0924 03:01:42.628999 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:01:42.629518 22685 raft_consensus.cc:348] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } }
I20260924 03:01:42.629659 22685 raft_consensus.cc:729] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18cc6a9ad9874d439cfbc151fcef2260, State: Initialized, Role: FOLLOWER
I20260924 03:01:42.629746 22685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } }
I20260924 03:01:42.630045 22685 sys_catalog.cc:564] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260 [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:42.630100 22659 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "124f912fb6db485e871ba041d41aa647" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "18cc6a9ad9874d439cfbc151fcef2260"
I20260924 03:01:42.630190 22659 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260 [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:42.630980 22659 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 124f912fb6db485e871ba041d41aa647 in term 1.
I20260924 03:01:42.631186 22658 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260 [term 1 FOLLOWER]: Refusing update from remote peer 124f912fb6db485e871ba041d41aa647: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:01:42.631421 22794 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [LEADER]: Connected to new peer: Peer: permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260924 03:01:42.633546 22825 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:01:42.633592 22825 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:01:42.633692 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 124f912fb6db485e871ba041d41aa647. Latest consensus state: current_term: 1 leader_uuid: "124f912fb6db485e871ba041d41aa647" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } } }
I20260924 03:01:42.633757 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:01:42.633798 22819 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "124f912fb6db485e871ba041d41aa647" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } } }
I20260924 03:01:42.633841 22819 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:01:42.634895 22820 mvcc.cc:205] Tried to move back new op lower bound from 7332736625117757440 to 7332736625019383808. Current Snapshot: MvccSnapshot[applied={T|T < 7332736625038688256}]
I20260924 03:01:42.637876 22794 sys_catalog.cc:455] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "124f912fb6db485e871ba041d41aa647" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } } }
I20260924 03:01:42.637960 22794 sys_catalog.cc:458] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647 [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:42.632827 22822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "124f912fb6db485e871ba041d41aa647" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "124f912fb6db485e871ba041d41aa647" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 45395 } } peers { permanent_uuid: "18cc6a9ad9874d439cfbc151fcef2260" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 36413 } } peers { permanent_uuid: "26aa13c97dc641249b3c47ce3829e2e9" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 32959 } } }
I20260924 03:01:42.638206 22822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260 [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:01:42.638652 22796 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 124f912fb6db485e871ba041d41aa647: Generated new TSK 0
I20260924 03:01:42.638825 22796 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:01:42.661798 22689 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:42.660196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41116 (local address 127.0.98.61:36413)
0924 03:01:42.660241 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.660242 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.661099 (+ 857us) server_negotiation.cc:407] Connection header received
0924 03:01:42.661227 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.661228 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.661239 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.661281 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.661713 (+ 432us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41116: BlockingRecv error: recv got EOF from 127.0.0.1:41116 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 03:01:42.662575 22613 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:42.660590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43882 (local address 127.0.98.62:45395)
0924 03:01:42.660628 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.660630 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.662024 (+ 1394us) server_negotiation.cc:407] Connection header received
0924 03:01:42.662115 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.662117 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.662127 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.662154 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.662488 (+ 334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43882: BlockingRecv error: recv got EOF from 127.0.0.1:43882 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 03:01:42.663256 22764 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:42.660873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37388 (local address 127.0.98.60:32959)
0924 03:01:42.660915 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.660917 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.662753 (+ 1836us) server_negotiation.cc:407] Connection header received
0924 03:01:42.662884 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.662885 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.662894 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.662921 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.663202 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37388: BlockingRecv error: recv got EOF from 127.0.0.1:37388 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 03:01:42.667815 22689 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:42.667110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41118 (local address 127.0.98.61:36413)
0924 03:01:42.667145 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.667146 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.667320 (+ 174us) server_negotiation.cc:407] Connection header received
0924 03:01:42.667417 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.667418 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.667436 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.667463 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.667756 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41118: BlockingRecv error: recv got EOF from 127.0.0.1:41118 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260924 03:01:42.670691 22689 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:42.669956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41128 (local address 127.0.98.61:36413)
0924 03:01:42.669987 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.669989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.670218 (+ 229us) server_negotiation.cc:407] Connection header received
0924 03:01:42.670302 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.670303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.670313 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.670339 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.670646 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41128: BlockingRecv error: recv got EOF from 127.0.0.1:41128 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 03:01:42.674319 22689 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:42.673631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41142 (local address 127.0.98.61:36413)
0924 03:01:42.673662 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.673664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.673891 (+ 227us) server_negotiation.cc:407] Connection header received
0924 03:01:42.673975 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.673975 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.673984 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.674011 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.674280 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41142: BlockingRecv error: recv got EOF from 127.0.0.1:41142 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 03:01:42.680055 22689 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:42.679343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41146 (local address 127.0.98.61:36413)
0924 03:01:42.679373 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.679375 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.679603 (+ 228us) server_negotiation.cc:407] Connection header received
0924 03:01:42.679684 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.679685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.679695 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.679720 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.680011 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41146: BlockingRecv error: recv got EOF from 127.0.0.1:41146 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 03:01:42.689978 22689 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:42.689147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41160 (local address 127.0.98.61:36413)
0924 03:01:42.689187 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.689189 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.689439 (+ 250us) server_negotiation.cc:407] Connection header received
0924 03:01:42.689530 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.689531 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.689541 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.689568 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.689913 (+ 345us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41160: BlockingRecv error: recv got EOF from 127.0.0.1:41160 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 03:01:42.708176 22689 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:42.707208 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41164 (local address 127.0.98.61:36413)
0924 03:01:42.707251 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.707253 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.707533 (+ 280us) server_negotiation.cc:407] Connection header received
0924 03:01:42.707635 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.707636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.707648 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.707695 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.708117 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41164: BlockingRecv error: recv got EOF from 127.0.0.1:41164 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 03:01:42.742353 22689 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:42.741443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41168 (local address 127.0.98.61:36413)
0924 03:01:42.741489 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.741491 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.741790 (+ 299us) server_negotiation.cc:407] Connection header received
0924 03:01:42.741889 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.741890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.741901 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.741929 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.742292 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41168: BlockingRecv error: recv got EOF from 127.0.0.1:41168 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 03:01:42.808115 22689 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:42.807530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41174 (local address 127.0.98.61:36413)
0924 03:01:42.807613 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.807615 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.807661 (+ 46us) server_negotiation.cc:407] Connection header received
0924 03:01:42.807710 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.807711 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.807720 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.807776 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.808054 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41174: BlockingRecv error: recv got EOF from 127.0.0.1:41174 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
W20260924 03:01:42.937820 22689 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:42.937148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41176 (local address 127.0.98.61:36413)
0924 03:01:42.937220 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:42.937223 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:42.937287 (+ 64us) server_negotiation.cc:407] Connection header received
0924 03:01:42.937366 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:42.937367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:42.937379 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:42.937428 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:42.937748 (+ 320us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41176: BlockingRecv error: recv got EOF from 127.0.0.1:41176 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
W20260924 03:01:43.195473 22689 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:43.194893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41180 (local address 127.0.98.61:36413)
0924 03:01:43.194968 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:43.194970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:43.194980 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:01:43.195029 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:43.195031 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:43.195043 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:43.195092 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:43.195411 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41180: BlockingRecv error: recv got EOF from 127.0.0.1:41180 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 03:01:43.560115 22778 catalog_manager.cc:1259] Loaded cluster ID: 3917fee02809401dabe9004f9267e32e
I20260924 03:01:43.560180 22778 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9: loading cluster ID for follower catalog manager: success
I20260924 03:01:43.561331 22778 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9: acquiring CA information for follower catalog manager: success
I20260924 03:01:43.561729 22778 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 26aa13c97dc641249b3c47ce3829e2e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:01:43.633989 22825 catalog_manager.cc:1259] Loaded cluster ID: 3917fee02809401dabe9004f9267e32e
I20260924 03:01:43.634061 22825 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260: loading cluster ID for follower catalog manager: success
I20260924 03:01:43.634984 22825 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260: acquiring CA information for follower catalog manager: success
I20260924 03:01:43.635288 22825 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 18cc6a9ad9874d439cfbc151fcef2260: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:01:43.710963 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:43.708470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32906 (local address 127.0.98.61:36413)
0924 03:01:43.708578 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:43.708580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:43.708636 (+ 56us) server_negotiation.cc:407] Connection header received
0924 03:01:43.708700 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:43.708702 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:43.708752 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:43.708835 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:01:43.708958 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:43.709464 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:43.710240 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:43.710433 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:43.710480 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:43.710641 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:01:43.710744 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:01:43.710802 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:43.710867 (+ 65us) server_negotiation.cc:299] Negotiation successful
0924 03:01:43.710884 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":58,"threads_started":1}
I20260924 03:01:43.715144 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:43.712696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.0.98.60:32959)
0924 03:01:43.712804 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:43.712806 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:43.712814 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:01:43.712853 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:43.712853 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:43.712890 (+ 37us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:43.712954 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:01:43.713077 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:43.713655 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:43.714372 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:43.714634 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:43.714689 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:43.714840 (+ 151us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:01:43.714945 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:01:43.715008 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:43.715045 (+ 37us) server_negotiation.cc:299] Negotiation successful
0924 03:01:43.715063 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":49,"threads_started":1}
I20260924 03:01:43.719182 22905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:43.716534 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.0.98.62:45395)
0924 03:01:43.716641 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:43.716643 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:43.716722 (+ 79us) server_negotiation.cc:407] Connection header received
0924 03:01:43.716988 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:43.716989 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:43.717040 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:43.717111 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:01:43.717203 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:43.717706 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:43.718402 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:43.718648 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:43.718696 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:43.718878 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:01:43.718977 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:01:43.719032 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:43.719082 (+ 50us) server_negotiation.cc:299] Negotiation successful
0924 03:01:43.719100 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":53,"threads_started":1}
I20260924 03:01:43.719558 22893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:43.716630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32920 (local address 127.0.98.61:36413)
0924 03:01:43.716666 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:43.716668 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:43.716730 (+ 62us) server_negotiation.cc:407] Connection header received
0924 03:01:43.716844 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:43.716846 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:43.716856 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:43.716900 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:01:43.717013 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:43.717734 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:43.718892 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:43.719128 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:43.719166 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:43.719349 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:01:43.719380 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:01:43.719425 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:43.719471 (+ 46us) server_negotiation.cc:299] Negotiation successful
0924 03:01:43.719492 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:01:43.719933 22899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:43.716762 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34518 (local address 127.0.98.60:32959)
0924 03:01:43.716809 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:43.716810 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:43.716816 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:01:43.716848 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:43.716848 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:43.716857 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:43.716900 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:01:43.717029 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:43.717737 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:43.718899 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:43.719159 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:43.719212 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:43.719549 (+ 337us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:01:43.719575 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:01:43.719748 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:43.719851 (+ 103us) server_negotiation.cc:299] Negotiation successful
0924 03:01:43.719870 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260924 03:01:43.725520 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 22542
I20260924 03:01:43.729861 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 22617
I20260924 03:01:43.733886 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 22692
2026-09-24T03:01:43Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2670 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 24 03:01:43 dist-test-slave-sb6p krb5kdc[22910](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 24 03:01:43 dist-test-slave-sb6p krb5kdc[22910](info): set up 2 sockets
Sep 24 03:01:43 dist-test-slave-sb6p krb5kdc[22910](info): commencing operation
krb5kdc: starting...
W20260924 03:01:44.674409 392 mini_kdc.cc:121] Time spent starting KDC: real 0.922s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/test-user.keytab.
Sep 24 03:01:44 dist-test-slave-sb6p krb5kdc[22910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-24T03:01:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-24T03:01:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.98.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
Entry for principal kudu/127.0.98.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab.
I20260924 03:01:44.783481 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46431
--webserver_interface=127.0.98.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46431,127.0.98.61:42549,127.0.98.60:33243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260924 03:01:44.861608 22923 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:01:44.861884 22923 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:44.861912 22923 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:44.861945 22923 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:44.863425 22923 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:44.863480 22923 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:44.863499 22923 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:44.863523 22923 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:44.863539 22923 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:44.865617 22923 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46431,127.0.98.61:42549,127.0.98.60:33243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/kudu_127.0.98.62.keytab
--principal=kudu/127.0.98.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.62:46431
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.98.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22923
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:44.865891 22923 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:44.866137 22923 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:44.871862 22928 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:44.871941 22929 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:44.871920 22931 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:44.872355 22923 server_base.cc:1034] running on GCE node
I20260924 03:01:44.872570 22923 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:44.872848 22923 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:44.873992 22923 hybrid_clock.cc:648] HybridClock initialized: now 1790218904873979 us; error 30 us; skew 500 ppm
Sep 24 03:01:44 dist-test-slave-sb6p krb5kdc[22910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:44.876302 22923 init.cc:377] Logged in from keytab as kudu/127.0.98.62@KRBTEST.COM (short username kudu)
I20260924 03:01:44.876705 22923 webserver.cc:460] Webserver started at http://127.0.98.62:40263/ using document root <none> and password file <none>
I20260924 03:01:44.876914 22923 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:44.876960 22923 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:44.877080 22923 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:44.878134 22923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-0/data/instance:
uuid: "d93d570105974eff97eb9c8ea5065ded"
format_stamp: "Formatted at 2026-09-24 03:01:44 on dist-test-slave-sb6p"
server_key: "0cb61e9c90e067018a8e8ea29dbe9a50"
server_key_iv: "c015cec330cbdd3c5901f8dc04cbd399"
server_key_version: "encryptionkey@0"
I20260924 03:01:44.878450 22923 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-0/wal/instance:
uuid: "d93d570105974eff97eb9c8ea5065ded"
format_stamp: "Formatted at 2026-09-24 03:01:44 on dist-test-slave-sb6p"
server_key: "0cb61e9c90e067018a8e8ea29dbe9a50"
server_key_iv: "c015cec330cbdd3c5901f8dc04cbd399"
server_key_version: "encryptionkey@0"
I20260924 03:01:44.879668 22923 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260924 03:01:44.880394 22938 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:44.880555 22923 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260924 03:01:44.880633 22923 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-0/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-0/wal
uuid: "d93d570105974eff97eb9c8ea5065ded"
format_stamp: "Formatted at 2026-09-24 03:01:44 on dist-test-slave-sb6p"
server_key: "0cb61e9c90e067018a8e8ea29dbe9a50"
server_key_iv: "c015cec330cbdd3c5901f8dc04cbd399"
server_key_version: "encryptionkey@0"
I20260924 03:01:44.880697 22923 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:44.890621 22923 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:44.892292 22923 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:44.892422 22923 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:44.896090 22923 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.62:46431
I20260924 03:01:44.896121 22990 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.62:46431 every 8 connection(s)
I20260924 03:01:44.896420 22923 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-0/data/info.pb
I20260924 03:01:44.897301 22991 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:44.897619 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 22923
I20260924 03:01:44.897709 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-0/wal/instance
I20260924 03:01:44.897866 392 external_mini_cluster.cc:1428] Setting key 269c34b6baca4d2ba0a4a488b794b07a
I20260924 03:01:44.898888 22991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.98.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.61@KRBTEST.COM" created.
Sep 24 03:01:44 dist-test-slave-sb6p krb5kdc[22910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:44.909812 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:44.899598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46431 (local address 127.0.0.1:40848)
0924 03:01:44.899764 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:44.899832 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:44.900285 (+ 453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:44.900532 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:44.900534 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:44.900803 (+ 269us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:44.900948 (+ 145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:44.900952 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:44.901916 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:44.901917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:44.902633 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:44.902636 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:44.903079 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:44.903080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:44.903141 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:44.903594 (+ 453us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:44.903602 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:44.906933 (+ 3331us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:44.908477 (+ 1544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:44.908483 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:44.908485 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:44.908673 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:44.908868 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:44.908868 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:44.908869 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:44.908948 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:44.909254 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:44.909259 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:44.909321 (+ 62us) client_negotiation.cc:769] Sending connection context
0924 03:01:44.909402 (+ 81us) client_negotiation.cc:240] Negotiation successful
0924 03:01:44.909632 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":97,"thread_start_us":78,"threads_started":1}
I20260924 03:01:44.909811 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:44.899612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40848 (local address 127.0.98.62:46431)
0924 03:01:44.899764 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:44.899773 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:44.899917 (+ 144us) server_negotiation.cc:407] Connection header received
0924 03:01:44.900429 (+ 512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:44.900431 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:44.900446 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:44.900520 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:44.901039 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:44.901834 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:44.902700 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:44.902994 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:44.903094 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:44.907009 (+ 3915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:44.907015 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:44.907016 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:44.907023 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:44.908405 (+ 1382us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:44.908739 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:44.908743 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:44.908743 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:44.908814 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:44.908998 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:44.908999 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:44.908999 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:44.909198 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:44.909269 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:44.909430 (+ 161us) server_negotiation.cc:299] Negotiation successful
0924 03:01:44.909580 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260924 03:01:44.910465 22991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:01:44.910820 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:44.910720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42549 (local address 127.0.0.1:37034)
0924 03:01:44.910754 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:44.910800 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260924 03:01:44.910934 22942 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42549: connect: Connection refused (error 111)
I20260924 03:01:44.911314 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:44.911242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42549 (local address 127.0.0.1:37038)
0924 03:01:44.911276 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:44.911300 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260924 03:01:44.911439 22991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42549: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42549: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/kudu_127.0.98.61.keytab.
Entry for principal kudu/127.0.98.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/kudu_127.0.98.61.keytab.
I20260924 03:01:44.919440 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42549
--webserver_interface=127.0.98.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46431,127.0.98.61:42549,127.0.98.60:33243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:01:44.960645 22991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } attempt: 1
I20260924 03:01:44.961099 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:44.960982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42549 (local address 127.0.0.1:37042)
0924 03:01:44.961035 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:44.961068 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260924 03:01:44.961458 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:44.961380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42549 (local address 127.0.0.1:37044)
0924 03:01:44.961419 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:44.961441 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260924 03:01:44.961555 22991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42549: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42549: connect: Connection refused (error 111)
W20260924 03:01:44.997315 22998 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:01:44.997527 22998 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:44.997550 22998 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:44.997576 22998 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:44.998719 22998 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:44.998757 22998 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:44.998776 22998 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:44.998792 22998 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:44.998807 22998 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:45.000274 22998 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46431,127.0.98.61:42549,127.0.98.60:33243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/kudu_127.0.98.61.keytab
--principal=kudu/127.0.98.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.61:42549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.98.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22998
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:45.000499 22998 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:45.000706 22998 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:45.005369 23003 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:45.005373 23004 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:45.005404 23006 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:45.005563 22998 server_base.cc:1034] running on GCE node
I20260924 03:01:45.005779 22998 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:45.006014 22998 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:45.007146 22998 hybrid_clock.cc:648] HybridClock initialized: now 1790218905007127 us; error 32 us; skew 500 ppm
Sep 24 03:01:45 dist-test-slave-sb6p krb5kdc[22910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:45.009632 22998 init.cc:377] Logged in from keytab as kudu/127.0.98.61@KRBTEST.COM (short username kudu)
I20260924 03:01:45.010076 22998 webserver.cc:460] Webserver started at http://127.0.98.61:41817/ using document root <none> and password file <none>
I20260924 03:01:45.010285 22998 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:45.010329 22998 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:45.010447 22998 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:45.011495 22998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-1/data/instance:
uuid: "569e3df64bdb4ca6b6009feae1a5607b"
format_stamp: "Formatted at 2026-09-24 03:01:45 on dist-test-slave-sb6p"
server_key: "fb40f3e2fd8b6c0c804a4d87e2b397c0"
server_key_iv: "1e2dddb8ebb3d96135a24b6dcd29dc1d"
server_key_version: "encryptionkey@0"
I20260924 03:01:45.011861 22998 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-1/wal/instance:
uuid: "569e3df64bdb4ca6b6009feae1a5607b"
format_stamp: "Formatted at 2026-09-24 03:01:45 on dist-test-slave-sb6p"
server_key: "fb40f3e2fd8b6c0c804a4d87e2b397c0"
server_key_iv: "1e2dddb8ebb3d96135a24b6dcd29dc1d"
server_key_version: "encryptionkey@0"
I20260924 03:01:45.013049 22998 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260924 03:01:45.013695 23013 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:45.013840 22998 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:45.013913 22998 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-1/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-1/wal
uuid: "569e3df64bdb4ca6b6009feae1a5607b"
format_stamp: "Formatted at 2026-09-24 03:01:45 on dist-test-slave-sb6p"
server_key: "fb40f3e2fd8b6c0c804a4d87e2b397c0"
server_key_iv: "1e2dddb8ebb3d96135a24b6dcd29dc1d"
server_key_version: "encryptionkey@0"
I20260924 03:01:45.013976 22998 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:45.024993 22998 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:45.026798 22998 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:45.026916 22998 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:45.029704 22991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } attempt: 2
I20260924 03:01:45.030113 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.029972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42549 (local address 127.0.0.1:37058)
0924 03:01:45.030030 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.030073 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260924 03:01:45.030440 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.030344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42549 (local address 127.0.0.1:37072)
0924 03:01:45.030388 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.030416 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42549: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260924 03:01:45.030603 22991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.61:42549: Network error: Client connection negotiation failed: client connection to 127.0.98.61:42549: connect: Connection refused (error 111)
I20260924 03:01:45.030772 22998 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.61:42549
I20260924 03:01:45.030833 23065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.61:42549 every 8 connection(s)
I20260924 03:01:45.031196 22998 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-1/data/info.pb
I20260924 03:01:45.032196 23066 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:45.033547 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 22998
I20260924 03:01:45.033653 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-1/wal/instance
I20260924 03:01:45.033842 392 external_mini_cluster.cc:1428] Setting key d16ad9c8d7a14626aa6067adc899bdea
I20260924 03:01:45.033726 23066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:01:45 dist-test-slave-sb6p krb5kdc[22910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.98.60@KRBTEST.COM; defaulting to no policy
I20260924 03:01:45.042317 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.034359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.0.98.62:46431)
0924 03:01:45.034396 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.034398 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.034655 (+ 257us) server_negotiation.cc:407] Connection header received
0924 03:01:45.035157 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.035158 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.035171 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.035233 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.035865 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.036433 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.037464 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.037670 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.037723 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.040022 (+ 2299us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:45.040024 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:45.040024 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:45.040030 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:45.041248 (+ 1218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.041578 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.041578 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.041579 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.041623 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.041838 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.041839 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.041839 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.041937 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:45.041982 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:45.042190 (+ 208us) server_negotiation.cc:299] Negotiation successful
0924 03:01:45.042250 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260924 03:01:45.042512 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.034369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46431 (local address 127.0.0.1:40856)
0924 03:01:45.034535 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.034573 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:45.035064 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:45.035359 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:45.035362 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:45.035595 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:45.035775 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:45.035779 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.036518 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.036519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:45.037388 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:45.037391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.037747 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.037748 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:45.037867 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.038344 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:45.038353 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:45.039934 (+ 1581us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:45.041317 (+ 1383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:45.041323 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:45.041325 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:45.041519 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:45.041676 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:45.041677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:45.041678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:45.041783 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:45.041981 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:45.041984 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:45.042093 (+ 109us) client_negotiation.cc:769] Sending connection context
0924 03:01:45.042177 (+ 84us) client_negotiation.cc:240] Negotiation successful
0924 03:01:45.042293 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.98.60@KRBTEST.COM" created.
I20260924 03:01:45.043072 23066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:01:45 dist-test-slave-sb6p krb5kdc[22910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:01:45.049815 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.043376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37076 (local address 127.0.98.61:42549)
0924 03:01:45.043524 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.043528 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.043545 (+ 17us) server_negotiation.cc:407] Connection header received
0924 03:01:45.043564 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.043566 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.043579 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.043647 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.043964 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.044629 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.045278 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.045584 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.045640 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.047420 (+ 1780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:45.047427 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:45.047427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:45.047433 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:45.048780 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.049126 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.049128 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.049128 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.049175 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.049340 (+ 165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.049341 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.049341 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.049459 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:45.049514 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:45.049643 (+ 129us) server_negotiation.cc:299] Negotiation successful
0924 03:01:45.049707 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":76,"threads_started":1}
I20260924 03:01:45.049897 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.043338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42549 (local address 127.0.0.1:37076)
0924 03:01:45.043376 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.043387 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:45.043430 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:45.043715 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:45.043716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:45.043846 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:45.043898 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:45.043900 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.044704 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.044704 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:45.045211 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:45.045213 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.045644 (+ 431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.045645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:45.045702 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.046069 (+ 367us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:45.046074 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:45.047346 (+ 1272us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:45.048863 (+ 1517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:45.048864 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:45.048864 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:45.049057 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:45.049228 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:45.049229 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:45.049229 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:45.049289 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:45.049510 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:45.049511 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:45.049554 (+ 43us) client_negotiation.cc:769] Sending connection context
0924 03:01:45.049621 (+ 67us) client_negotiation.cc:240] Negotiation successful
0924 03:01:45.049765 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22,"mutex_wait_us":48}
I20260924 03:01:45.050415 23066 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:01:45.050766 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.050646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33243 (local address 127.0.0.1:34816)
0924 03:01:45.050709 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.050749 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260924 03:01:45.050884 23017 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33243: connect: Connection refused (error 111)
I20260924 03:01:45.051257 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.051179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33243 (local address 127.0.0.1:34822)
0924 03:01:45.051210 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.051234 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260924 03:01:45.051357 23066 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33243: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33243: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/kudu_127.0.98.60.keytab.
Entry for principal kudu/127.0.98.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/kudu_127.0.98.60.keytab.
I20260924 03:01:45.056782 392 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskPoI4uN/build/release/bin/kudu
/tmp/dist-test-taskPoI4uN/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33243
--webserver_interface=127.0.98.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.98.20:39073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.98.62:46431,127.0.98.61:42549,127.0.98.60:33243
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260924 03:01:45.100580 23066 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } attempt: 1
I20260924 03:01:45.101045 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.100926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33243 (local address 127.0.0.1:34830)
0924 03:01:45.100984 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.101016 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 03:01:45.101363 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.101271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33243 (local address 127.0.0.1:34840)
0924 03:01:45.101322 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.101344 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260924 03:01:45.101450 23066 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33243: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33243: connect: Connection refused (error 111)
I20260924 03:01:45.121747 22991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } attempt: 3
Sep 24 03:01:45 dist-test-slave-sb6p krb5kdc[22910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:01:45.128675 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.122056 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42549 (local address 127.0.0.1:37092)
0924 03:01:45.122113 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.122127 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:45.122175 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:45.122333 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:45.122334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:45.122467 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:45.122549 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:45.122551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.123496 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.123496 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:45.124012 (+ 516us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:45.124013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.124468 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.124469 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:45.124529 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.124962 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:45.124966 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:45.126265 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:45.127848 (+ 1583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:45.127848 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:45.127848 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:45.128000 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:45.128183 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:45.128183 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:45.128184 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:45.128226 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:45.128445 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:45.128445 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:45.128482 (+ 37us) client_negotiation.cc:769] Sending connection context
0924 03:01:45.128525 (+ 43us) client_negotiation.cc:240] Negotiation successful
0924 03:01:45.128596 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 03:01:45.128736 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.122135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37092 (local address 127.0.98.61:42549)
0924 03:01:45.122196 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.122198 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.122208 (+ 10us) server_negotiation.cc:407] Connection header received
0924 03:01:45.122235 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.122236 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.122249 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.122318 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.122605 (+ 287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.123414 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.124091 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.124401 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.124471 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.126361 (+ 1890us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:45.126362 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:45.126362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:45.126370 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:45.127773 (+ 1403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.128064 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.128065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.128066 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.128128 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.128261 (+ 133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.128262 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.128262 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.128393 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:45.128458 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:45.128550 (+ 92us) server_negotiation.cc:299] Negotiation successful
0924 03:01:45.128639 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260924 03:01:45.129035 22991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } has no permanent_uuid. Determining permanent_uuid...
I20260924 03:01:45.129343 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.129230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33243 (local address 127.0.0.1:34844)
0924 03:01:45.129279 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.129315 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260924 03:01:45.129595 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.129530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33243 (local address 127.0.0.1:34858)
0924 03:01:45.129554 (+ 24us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.129577 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
W20260924 03:01:45.129680 22991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33243: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33243: connect: Connection refused (error 111)
W20260924 03:01:45.136158 23073 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260924 03:01:45.136345 23073 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260924 03:01:45.136364 23073 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260924 03:01:45.136399 23073 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260924 03:01:45.137490 23073 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260924 03:01:45.137526 23073 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260924 03:01:45.137538 23073 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260924 03:01:45.137549 23073 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260924 03:01:45.137562 23073 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260924 03:01:45.139066 23073 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.98.20:39073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.98.62:46431,127.0.98.61:42549,127.0.98.60:33243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/krb5kdc/kudu_127.0.98.60.keytab
--principal=kudu/127.0.98.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.98.60:33243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.98.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23073
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 24 Sep 2026 02:05:26 UTC on ccycloud.quasar-acuapc.root.comops.site
build id 4669247
I20260924 03:01:45.139295 23073 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260924 03:01:45.139487 23073 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260924 03:01:45.144052 23079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:45.144126 23073 server_base.cc:1034] running on GCE node
W20260924 03:01:45.144068 23078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260924 03:01:45.144057 23081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260924 03:01:45.144471 23073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260924 03:01:45.144775 23073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260924 03:01:45.145951 23073 hybrid_clock.cc:648] HybridClock initialized: now 1790218905145937 us; error 28 us; skew 500 ppm
Sep 24 03:01:45 dist-test-slave-sb6p krb5kdc[22910](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260924 03:01:45.148337 23073 init.cc:377] Logged in from keytab as kudu/127.0.98.60@KRBTEST.COM (short username kudu)
I20260924 03:01:45.148777 23073 webserver.cc:460] Webserver started at http://127.0.98.60:35311/ using document root <none> and password file <none>
I20260924 03:01:45.148947 23073 fs_manager.cc:359] Metadata directory not provided
I20260924 03:01:45.148984 23073 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260924 03:01:45.149070 23073 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260924 03:01:45.150043 23073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-2/data/instance:
uuid: "9e74069e7ae3413eb24e0eb8b92ec6da"
format_stamp: "Formatted at 2026-09-24 03:01:45 on dist-test-slave-sb6p"
server_key: "088e7f102090772294f5bb43e219a171"
server_key_iv: "b22de3fb6de0c042f29c51a6baee204e"
server_key_version: "encryptionkey@0"
I20260924 03:01:45.150317 23073 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-2/wal/instance:
uuid: "9e74069e7ae3413eb24e0eb8b92ec6da"
format_stamp: "Formatted at 2026-09-24 03:01:45 on dist-test-slave-sb6p"
server_key: "088e7f102090772294f5bb43e219a171"
server_key_iv: "b22de3fb6de0c042f29c51a6baee204e"
server_key_version: "encryptionkey@0"
I20260924 03:01:45.151549 23073 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260924 03:01:45.152253 23088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260924 03:01:45.152422 23073 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260924 03:01:45.152513 23073 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-2/data,/tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-2/wal
uuid: "9e74069e7ae3413eb24e0eb8b92ec6da"
format_stamp: "Formatted at 2026-09-24 03:01:45 on dist-test-slave-sb6p"
server_key: "088e7f102090772294f5bb43e219a171"
server_key_iv: "b22de3fb6de0c042f29c51a6baee204e"
server_key_version: "encryptionkey@0"
I20260924 03:01:45.152592 23073 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260924 03:01:45.160825 22991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } attempt: 1
I20260924 03:01:45.161240 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.161091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33243 (local address 127.0.0.1:34874)
0924 03:01:45.161157 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.161203 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260924 03:01:45.161530 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.161459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33243 (local address 127.0.0.1:34890)
0924 03:01:45.161492 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.161513 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260924 03:01:45.161609 22991 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33243: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33243: connect: Connection refused (error 111)
I20260924 03:01:45.167068 23073 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260924 03:01:45.168732 23073 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260924 03:01:45.168849 23073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260924 03:01:45.169574 23066 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } attempt: 2
I20260924 03:01:45.169914 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.169778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33243 (local address 127.0.0.1:34902)
0924 03:01:45.169839 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.169885 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260924 03:01:45.170248 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.170163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33243 (local address 127.0.0.1:34918)
0924 03:01:45.170205 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.170229 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33243: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":14}
W20260924 03:01:45.170320 23066 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.0.98.60:33243: Network error: Client connection negotiation failed: client connection to 127.0.98.60:33243: connect: Connection refused (error 111)
I20260924 03:01:45.172924 23073 rpc_server.cc:307] RPC server started. Bound to: 127.0.98.60:33243
I20260924 03:01:45.172952 23140 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.98.60:33243 every 8 connection(s)
I20260924 03:01:45.173257 23073 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-2/data/info.pb
I20260924 03:01:45.174173 23141 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260924 03:01:45.175645 23141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:01:45 dist-test-slave-sb6p krb5kdc[22910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
I20260924 03:01:45.180847 392 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskPoI4uN/build/release/bin/kudu as pid 23073
I20260924 03:01:45.180946 392 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskPoI4uN/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790218647334776-392-0/minicluster-data/master-2/wal/instance
I20260924 03:01:45.181133 392 external_mini_cluster.cc:1428] Setting key 22a4553a0aba5d08bedf9169c8338b5b
I20260924 03:01:45.184018 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.176245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40866 (local address 127.0.98.62:46431)
0924 03:01:45.176284 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.176285 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.176469 (+ 184us) server_negotiation.cc:407] Connection header received
0924 03:01:45.176919 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.176920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.176928 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.176980 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.177535 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.177990 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.179038 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.179223 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.179280 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.181334 (+ 2054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:45.181335 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:45.181335 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:45.181343 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:45.182679 (+ 1336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.183127 (+ 448us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.183127 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.183128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.183174 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.183403 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.183403 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.183403 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.183508 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:45.183561 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:45.183855 (+ 294us) server_negotiation.cc:299] Negotiation successful
0924 03:01:45.183929 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:01:45.184187 23143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.176215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.62:46431 (local address 127.0.0.1:40866)
0924 03:01:45.176364 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.176404 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:45.176842 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:45.177041 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:45.177044 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:45.177284 (+ 240us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:45.177474 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:45.177478 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.178068 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.178069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:45.178947 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:45.178950 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.179330 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.179331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:45.179444 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.179858 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:45.179872 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:45.181240 (+ 1368us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:45.182759 (+ 1519us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:45.182767 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:45.182769 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:45.183064 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:45.183229 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:45.183230 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:45.183230 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:45.183352 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:45.183568 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:45.183575 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:45.183715 (+ 140us) client_negotiation.cc:769] Sending connection context
0924 03:01:45.183824 (+ 109us) client_negotiation.cc:240] Negotiation successful
0924 03:01:45.183977 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":63,"threads_started":1}
I20260924 03:01:45.184988 23141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:01:45 dist-test-slave-sb6p krb5kdc[22910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
Sep 24 03:01:45 dist-test-slave-sb6p krb5kdc[22910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:01:45.189265 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.181509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34934 (local address 127.0.98.60:33243)
0924 03:01:45.181653 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.181658 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.181753 (+ 95us) server_negotiation.cc:407] Connection header received
0924 03:01:45.181802 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.181805 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.181819 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.181884 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.182449 (+ 565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.183497 (+ 1048us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.184290 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.184614 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.184674 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.186776 (+ 2102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:45.186784 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:45.186784 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:45.186797 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:45.188137 (+ 1340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.188445 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.188448 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.188449 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.188503 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.188680 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.188681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.188682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.188832 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:45.188976 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:45.189054 (+ 78us) server_negotiation.cc:299] Negotiation successful
0924 03:01:45.189124 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":85,"threads_started":1}
I20260924 03:01:45.191174 23143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.185238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.61:42549 (local address 127.0.0.1:37096)
0924 03:01:45.185286 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.185295 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:45.185324 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:45.185472 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:45.185473 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:45.185595 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:45.185645 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:45.185647 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.186226 (+ 579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.186227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:45.186839 (+ 612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:45.186842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.187260 (+ 418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.187261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:45.187327 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.187707 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:45.187712 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:45.188823 (+ 1111us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:45.190314 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:45.190315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:45.190315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:45.190464 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:45.190640 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:45.190640 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:45.190641 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:45.190693 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:45.190905 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:45.190905 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:45.190945 (+ 40us) client_negotiation.cc:769] Sending connection context
0924 03:01:45.191003 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 03:01:45.191084 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260924 03:01:45.191180 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.185322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37096 (local address 127.0.98.61:42549)
0924 03:01:45.185380 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.185382 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.185390 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:01:45.185413 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.185414 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.185424 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.185465 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.185707 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.186160 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.186911 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.187202 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.187262 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.188895 (+ 1633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:45.188897 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:45.188897 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:45.188903 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:45.190242 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.190536 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.190537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.190537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.190590 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.190743 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.190743 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.190744 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.190854 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:45.190906 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:45.191030 (+ 124us) server_negotiation.cc:299] Negotiation successful
0924 03:01:45.191095 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260924 03:01:45.191602 23141 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } has no permanent_uuid. Determining permanent_uuid...
Sep 24 03:01:45 dist-test-slave-sb6p krb5kdc[22910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.62@KRBTEST.COM
Sep 24 03:01:45 dist-test-slave-sb6p krb5kdc[22910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.60@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:01:45.196632 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.190023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40880 (local address 127.0.98.62:46431)
0924 03:01:45.190064 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.190066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.190074 (+ 8us) server_negotiation.cc:407] Connection header received
0924 03:01:45.190112 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.190113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.190122 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.190175 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.190584 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.191044 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.191803 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.192017 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.192069 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.194270 (+ 2201us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:45.194271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:45.194271 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:45.194277 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:45.195648 (+ 1371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.195962 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.195963 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.195963 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.196017 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.196181 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.196181 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.196181 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.196270 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:45.196408 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:45.196471 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 03:01:45.196544 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260924 03:01:45.198338 23143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.191824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33243 (local address 127.0.0.1:34946)
0924 03:01:45.192108 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.192117 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:45.192149 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:45.192402 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:45.192403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:45.192501 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:45.192528 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:45.192528 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.193368 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.193369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:45.194057 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:45.194059 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.194448 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.194449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:45.194503 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.194868 (+ 365us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:45.194873 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:45.195954 (+ 1081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:45.197483 (+ 1529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:45.197484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:45.197484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:45.197685 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:45.197846 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:45.197846 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:45.197847 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:45.197893 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:45.198122 (+ 229us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:45.198123 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:45.198164 (+ 41us) client_negotiation.cc:769] Sending connection context
0924 03:01:45.198214 (+ 50us) client_negotiation.cc:240] Negotiation successful
0924 03:01:45.198281 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":271}
I20260924 03:01:45.198431 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.191851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.0.98.60:33243)
0924 03:01:45.191882 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.191884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.192156 (+ 272us) server_negotiation.cc:407] Connection header received
0924 03:01:45.192178 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.192179 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.192189 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.192243 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.192593 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.193297 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.194126 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.194386 (+ 260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.194431 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.196028 (+ 1597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:45.196029 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:45.196029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:45.196034 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:45.197420 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.197746 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.197747 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.197747 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.197799 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.197938 (+ 139us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.197939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.197939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.198052 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:45.198115 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:45.198246 (+ 131us) server_negotiation.cc:299] Negotiation successful
0924 03:01:45.198352 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":19,"server-negotiator.queue_time_us":14}
I20260924 03:01:45.199942 23141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da: Bootstrap starting.
I20260924 03:01:45.200909 23141 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da: Neither blocks nor log segments found. Creating new log.
Sep 24 03:01:45 dist-test-slave-sb6p krb5kdc[22910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218904, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.98.61@KRBTEST.COM
I20260924 03:01:45.201170 23141 log.cc:824] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:45.201960 23141 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da: No bootstrap required, opened a new log
I20260924 03:01:45.203637 23141 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } }
I20260924 03:01:45.203809 23141 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e74069e7ae3413eb24e0eb8b92ec6da, State: Initialized, Role: FOLLOWER
I20260924 03:01:45.203905 23141 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } }
I20260924 03:01:45.203976 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.197172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37098 (local address 127.0.98.61:42549)
0924 03:01:45.197206 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.197208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.197240 (+ 32us) server_negotiation.cc:407] Connection header received
0924 03:01:45.197259 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.197259 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.197268 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.197321 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.197772 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.198422 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.199175 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.199359 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.199409 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.201631 (+ 2222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:45.201632 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:45.201632 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:45.201639 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:45.202952 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.203309 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.203309 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.203310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.203357 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.203528 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.203529 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.203529 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.203618 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:45.203735 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:45.203821 (+ 86us) server_negotiation.cc:299] Negotiation successful
0924 03:01:45.203890 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260924 03:01:45.204213 23141 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:45.204169 23147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } } }
I20260924 03:01:45.204267 23147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:01:45.205421 392 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
W20260924 03:01:45.206007 23158 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260924 03:01:45.206089 23158 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:01:45.206257 22965 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d93d570105974eff97eb9c8ea5065ded"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40880
I20260924 03:01:45.206269 23073 master_runner.cc:186] Error getting master registration for 127.0.98.62:46431: OK, instance_id { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" instance_seqno: 1790218904895067 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:01:45.208381 23073 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:01:45.236744 22991 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } attempt: 2
Sep 24 03:01:45 dist-test-slave-sb6p krb5kdc[22910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.62@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:01:45.244549 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.237161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.0.98.60:33243)
0924 03:01:45.237227 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.237230 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.237242 (+ 12us) server_negotiation.cc:407] Connection header received
0924 03:01:45.237272 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.237273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.237284 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.237342 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.237594 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.238340 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.239340 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.239661 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.239734 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.241859 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:45.241861 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:45.241861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:45.241868 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:45.243368 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.243745 (+ 377us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.243746 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.243747 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.243804 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.243973 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.243974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.243974 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.244158 (+ 184us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:45.244219 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:45.244367 (+ 148us) server_negotiation.cc:299] Negotiation successful
0924 03:01:45.244447 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260924 03:01:45.244552 22995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.237094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33243 (local address 127.0.0.1:34956)
0924 03:01:45.237155 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.237167 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:45.237231 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:45.237349 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:45.237349 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:45.237492 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:45.237537 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:45.237538 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.238410 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.238411 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:45.239252 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:45.239255 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.239731 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.239732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:45.239786 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.240275 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:45.240280 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:45.241758 (+ 1478us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:45.243445 (+ 1687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:45.243446 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:45.243447 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:45.243676 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:45.243859 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:45.243860 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:45.243860 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:45.243922 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:45.244214 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:45.244215 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:45.244271 (+ 56us) client_negotiation.cc:769] Sending connection context
0924 03:01:45.244337 (+ 66us) client_negotiation.cc:240] Negotiation successful
0924 03:01:45.244444 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260924 03:01:45.246290 22991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded: Bootstrap starting.
I20260924 03:01:45.247073 22991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:45.247385 22991 log.cc:824] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:45.248263 22991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded: No bootstrap required, opened a new log
I20260924 03:01:45.249663 22991 raft_consensus.cc:348] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } }
I20260924 03:01:45.249815 22991 raft_consensus.cc:729] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93d570105974eff97eb9c8ea5065ded, State: Initialized, Role: FOLLOWER
I20260924 03:01:45.249897 22991 consensus_queue.cc:260] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } }
I20260924 03:01:45.250209 22991 sys_catalog.cc:564] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:45.250238 22965 raft_consensus.cc:482] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260924 03:01:45.250329 22965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:45.251087 22965 raft_consensus.cc:504] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } }
I20260924 03:01:45.251318 22965 leader_election.cc:288] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [CANDIDATE]: Term 1 election: Requested vote from peers 569e3df64bdb4ca6b6009feae1a5607b (127.0.98.61:42549), 9e74069e7ae3413eb24e0eb8b92ec6da (127.0.98.60:33243)
I20260924 03:01:45.250169 23161 sys_catalog.cc:455] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } } }
I20260924 03:01:45.251408 23161 sys_catalog.cc:458] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:01:45.251873 23115 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d93d570105974eff97eb9c8ea5065ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da"
I20260924 03:01:45.251992 23115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:45.252846 23115 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d93d570105974eff97eb9c8ea5065ded in term 1.
I20260924 03:01:45.253075 22942 leader_election.cc:302] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9e74069e7ae3413eb24e0eb8b92ec6da, d93d570105974eff97eb9c8ea5065ded; no voters:
I20260924 03:01:45.253199 23161 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [term 1 FOLLOWER]: Leader election won for term 1
I20260924 03:01:45.253326 23161 raft_consensus.cc:686] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [term 1 LEADER]: Becoming Leader. State: Replica: d93d570105974eff97eb9c8ea5065ded, State: Running, Role: LEADER
I20260924 03:01:45.253430 23161 consensus_queue.cc:237] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } }
I20260924 03:01:45.255386 22923 master_runner.cc:186] Error getting master registration for 127.0.98.61:42549: OK, instance_id { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" instance_seqno: 1790218905029525 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260924 03:01:45.257642 22923 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260924 03:01:45.257817 23175 sys_catalog.cc:455] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [sys.catalog]: SysCatalogTable state changed. Reason: New leader d93d570105974eff97eb9c8ea5065ded. Latest consensus state: current_term: 1 leader_uuid: "d93d570105974eff97eb9c8ea5065ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } } }
I20260924 03:01:45.257951 23175 sys_catalog.cc:458] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:45.260370 22994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.252493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40888 (local address 127.0.98.62:46431)
0924 03:01:45.252528 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.252530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.252537 (+ 7us) server_negotiation.cc:407] Connection header received
0924 03:01:45.252579 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.252580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.252590 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.252633 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.253744 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.254423 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.255432 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.255697 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.255734 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.257206 (+ 1472us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:45.257207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:45.257208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:45.257215 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:45.258845 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.259365 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.259366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.259367 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.259435 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.259705 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.259706 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.259707 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.259844 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:45.259912 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:45.260193 (+ 281us) server_negotiation.cc:299] Negotiation successful
0924 03:01:45.260268 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:01:45.260985 23182 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260924 03:01:45.261389 23066 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } attempt: 3
I20260924 03:01:45.262241 23182 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260924 03:01:45.262394 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.253158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34970 (local address 127.0.98.60:33243)
0924 03:01:45.253191 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.253193 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.254326 (+ 1133us) server_negotiation.cc:407] Connection header received
0924 03:01:45.254417 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.254417 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.254427 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.254453 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.255229 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.255965 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.257258 (+ 1293us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.257804 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.257846 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.259449 (+ 1603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:45.259450 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:45.259450 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:45.259457 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:45.260983 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.261595 (+ 612us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.261596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.261597 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.261667 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.261852 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.261853 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.261853 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.261970 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:45.262035 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:45.262230 (+ 195us) server_negotiation.cc:299] Negotiation successful
0924 03:01:45.262303 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260924 03:01:45.265576 23070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.252735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37104 (local address 127.0.98.61:42549)
0924 03:01:45.252779 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.252780 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.253730 (+ 950us) server_negotiation.cc:407] Connection header received
0924 03:01:45.253866 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.253867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.253887 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.253917 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.258735 (+ 4818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.259403 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.260279 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.260559 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.260597 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.262900 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:45.262902 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:45.262902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:45.262909 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:45.264388 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.264750 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.264751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.264751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.264817 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.265011 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.265012 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.265013 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.265135 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:45.265213 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:45.265413 (+ 200us) server_negotiation.cc:299] Negotiation successful
0924 03:01:45.265487 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
Sep 24 03:01:45 dist-test-slave-sb6p krb5kdc[22910](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790218905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.98.61@KRBTEST.COM for kudu/127.0.98.60@KRBTEST.COM
I20260924 03:01:45.271942 23068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.261634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.98.60:33243 (local address 127.0.0.1:34972)
0924 03:01:45.261783 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0924 03:01:45.261806 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0924 03:01:45.261848 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0924 03:01:45.262642 (+ 794us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0924 03:01:45.262643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0924 03:01:45.262798 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0924 03:01:45.262835 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:45.262837 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.264140 (+ 1303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.264141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:45.265316 (+ 1175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0924 03:01:45.265319 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.265802 (+ 483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.265803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0924 03:01:45.265857 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.266207 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0924 03:01:45.266212 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0924 03:01:45.268150 (+ 1938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0924 03:01:45.270009 (+ 1859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:45.270010 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:45.270010 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:45.270225 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:45.271299 (+ 1074us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0924 03:01:45.271300 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0924 03:01:45.271300 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0924 03:01:45.271366 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0924 03:01:45.271631 (+ 265us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0924 03:01:45.271632 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0924 03:01:45.271677 (+ 45us) client_negotiation.cc:769] Sending connection context
0924 03:01:45.271735 (+ 58us) client_negotiation.cc:240] Negotiation successful
0924 03:01:45.271842 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132}
I20260924 03:01:45.272214 23145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:45.261667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34972 (local address 127.0.98.60:33243)
0924 03:01:45.262467 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.262468 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.262474 (+ 6us) server_negotiation.cc:407] Connection header received
0924 03:01:45.262490 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.262491 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.262500 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.262599 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.262911 (+ 312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.264052 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.265408 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:45.265701 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:45.265756 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:45.268240 (+ 2484us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0924 03:01:45.268241 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0924 03:01:45.268241 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0924 03:01:45.268247 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0924 03:01:45.269922 (+ 1675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.271156 (+ 1234us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.271157 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.271157 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.271224 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0924 03:01:45.271429 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0924 03:01:45.271430 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0924 03:01:45.271430 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0924 03:01:45.271553 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0924 03:01:45.271607 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:45.272062 (+ 455us) server_negotiation.cc:299] Negotiation successful
0924 03:01:45.272133 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":79,"threads_started":1}
I20260924 03:01:45.273360 23066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b: Bootstrap starting.
I20260924 03:01:45.274034 23066 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b: Neither blocks nor log segments found. Creating new log.
I20260924 03:01:45.274322 23066 log.cc:824] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b: Log is configured to *not* fsync() on all Append() calls
I20260924 03:01:45.275259 23066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b: No bootstrap required, opened a new log
I20260924 03:01:45.275470 23115 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da [term 1 FOLLOWER]: Refusing update from remote peer d93d570105974eff97eb9c8ea5065ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:01:45.275825 23175 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:01:45.277221 23066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } }
I20260924 03:01:45.277352 23066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 569e3df64bdb4ca6b6009feae1a5607b, State: Initialized, Role: FOLLOWER
I20260924 03:01:45.277428 23066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } }
I20260924 03:01:45.277755 23066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b [sys.catalog]: configured and running, proceeding with master startup.
I20260924 03:01:45.277850 23039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b [term 0 FOLLOWER]: Advancing to term 1
I20260924 03:01:45.278025 23147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da [sys.catalog]: SysCatalogTable state changed. Reason: New leader d93d570105974eff97eb9c8ea5065ded. Latest consensus state: current_term: 1 leader_uuid: "d93d570105974eff97eb9c8ea5065ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } } }
I20260924 03:01:45.278105 23147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:01:45.278568 23040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d93d570105974eff97eb9c8ea5065ded" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "569e3df64bdb4ca6b6009feae1a5607b"
I20260924 03:01:45.278746 23039 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b [term 1 FOLLOWER]: Refusing update from remote peer d93d570105974eff97eb9c8ea5065ded: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260924 03:01:45.279047 23175 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [LEADER]: Connected to new peer: Peer: permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260924 03:01:45.279662 23040 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d93d570105974eff97eb9c8ea5065ded in term 1.
I20260924 03:01:45.279633 23175 sys_catalog.cc:455] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d93d570105974eff97eb9c8ea5065ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } } }
I20260924 03:01:45.279716 23175 sys_catalog.cc:458] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:45.279847 23185 mvcc.cc:205] Tried to move back new op lower bound from 7332736635995623424 to 7332736635918807040. Current Snapshot: MvccSnapshot[applied={T|T < 7332736635995623424}]
I20260924 03:01:45.280004 23147 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d93d570105974eff97eb9c8ea5065ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } } }
I20260924 03:01:45.280076 23147 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:01:45.280440 23182 catalog_manager.cc:1347] Generated new cluster ID: e23d85fd39494b94b29205d6b231a9f7
I20260924 03:01:45.280483 23182 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260924 03:01:45.281081 23161 sys_catalog.cc:455] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d93d570105974eff97eb9c8ea5065ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } } }
I20260924 03:01:45.281175 23161 sys_catalog.cc:458] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded [sys.catalog]: This master's current role is: LEADER
I20260924 03:01:45.281706 23196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d93d570105974eff97eb9c8ea5065ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } } }
I20260924 03:01:45.281852 23196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:01:45.282367 23197 mvcc.cc:205] Tried to move back new op lower bound from 7332736635995623424 to 7332736635918807040. Current Snapshot: MvccSnapshot[applied={T|T < 7332736635995623424}]
I20260924 03:01:45.283106 23204 catalog_manager.cc:1259] Loaded cluster ID: e23d85fd39494b94b29205d6b231a9f7
I20260924 03:01:45.283197 23204 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b: loading cluster ID for follower catalog manager: success
W20260924 03:01:45.283334 23204 catalog_manager.cc:1576] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260924 03:01:45.283407 23204 catalog_manager.cc:873] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260924 03:01:45.283740 23196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b [sys.catalog]: SysCatalogTable state changed. Reason: New leader d93d570105974eff97eb9c8ea5065ded. Latest consensus state: current_term: 1 leader_uuid: "d93d570105974eff97eb9c8ea5065ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } } }
I20260924 03:01:45.283816 23196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:01:45.283849 23196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d93d570105974eff97eb9c8ea5065ded" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d93d570105974eff97eb9c8ea5065ded" member_type: VOTER last_known_addr { host: "127.0.98.62" port: 46431 } } peers { permanent_uuid: "569e3df64bdb4ca6b6009feae1a5607b" member_type: VOTER last_known_addr { host: "127.0.98.61" port: 42549 } } peers { permanent_uuid: "9e74069e7ae3413eb24e0eb8b92ec6da" member_type: VOTER last_known_addr { host: "127.0.98.60" port: 33243 } } }
I20260924 03:01:45.283888 23196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b [sys.catalog]: This master's current role is: FOLLOWER
I20260924 03:01:45.294044 23187 mvcc.cc:205] Tried to move back new op lower bound from 7332736635995623424 to 7332736635918807040. Current Snapshot: MvccSnapshot[applied={T|T < 7332736635995623424}]
I20260924 03:01:45.298588 23182 catalog_manager.cc:1370] Generated new certificate authority record
I20260924 03:01:45.299345 23182 catalog_manager.cc:1504] Loading token signing keys...
I20260924 03:01:45.310607 23182 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d93d570105974eff97eb9c8ea5065ded: Generated new TSK 0
I20260924 03:01:45.310881 23182 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260924 03:01:45.337083 23070 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:45.335501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37112 (local address 127.0.98.61:42549)
0924 03:01:45.335545 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.335547 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.336450 (+ 903us) server_negotiation.cc:407] Connection header received
0924 03:01:45.336553 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.336554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.336565 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.336594 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.337007 (+ 413us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37112: BlockingRecv error: recv got EOF from 127.0.0.1:37112 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260924 03:01:45.338043 22994 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:45.336176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40900 (local address 127.0.98.62:46431)
0924 03:01:45.336214 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.336216 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.337410 (+ 1194us) server_negotiation.cc:407] Connection header received
0924 03:01:45.337543 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.337544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.337555 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.337582 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.337982 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40900: BlockingRecv error: recv got EOF from 127.0.0.1:40900 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260924 03:01:45.338542 23145 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:45.335848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34974 (local address 127.0.98.60:33243)
0924 03:01:45.335885 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.335887 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.337344 (+ 1457us) server_negotiation.cc:407] Connection header received
0924 03:01:45.337641 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.337642 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.337653 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.337679 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.338463 (+ 784us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34974: BlockingRecv error: recv got EOF from 127.0.0.1:34974 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260924 03:01:45.340953 23070 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:45.340048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37122 (local address 127.0.98.61:42549)
0924 03:01:45.340082 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.340084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.340309 (+ 225us) server_negotiation.cc:407] Connection header received
0924 03:01:45.340412 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.340412 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.340430 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.340458 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.340911 (+ 453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37122: BlockingRecv error: recv got EOF from 127.0.0.1:37122 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 03:01:45.343778 23070 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:45.343060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37128 (local address 127.0.98.61:42549)
0924 03:01:45.343093 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.343094 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.343324 (+ 230us) server_negotiation.cc:407] Connection header received
0924 03:01:45.343415 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.343415 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.343425 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.343452 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.343734 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37128: BlockingRecv error: recv got EOF from 127.0.0.1:37128 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260924 03:01:45.347448 23070 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:45.346750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37142 (local address 127.0.98.61:42549)
0924 03:01:45.346779 (+ 29us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.346781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.346996 (+ 215us) server_negotiation.cc:407] Connection header received
0924 03:01:45.347083 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.347084 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.347094 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.347120 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.347406 (+ 286us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37142: BlockingRecv error: recv got EOF from 127.0.0.1:37142 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 03:01:45.353104 23070 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:45.352423 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37154 (local address 127.0.98.61:42549)
0924 03:01:45.352453 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.352455 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.352673 (+ 218us) server_negotiation.cc:407] Connection header received
0924 03:01:45.352751 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.352752 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.352761 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.352785 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.353062 (+ 277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37154: BlockingRecv error: recv got EOF from 127.0.0.1:37154 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260924 03:01:45.363026 23070 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:45.362142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37170 (local address 127.0.98.61:42549)
0924 03:01:45.362180 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.362182 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.362454 (+ 272us) server_negotiation.cc:407] Connection header received
0924 03:01:45.362585 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.362586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.362598 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.362624 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.362974 (+ 350us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37170: BlockingRecv error: recv got EOF from 127.0.0.1:37170 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260924 03:01:45.381330 23070 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:45.380275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37174 (local address 127.0.98.61:42549)
0924 03:01:45.380347 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.380349 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.380710 (+ 361us) server_negotiation.cc:407] Connection header received
0924 03:01:45.380813 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.380814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.380827 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.380856 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.381254 (+ 398us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37174: BlockingRecv error: recv got EOF from 127.0.0.1:37174 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260924 03:01:45.415493 23070 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:45.414790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.0.98.61:42549)
0924 03:01:45.414837 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.414839 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.414878 (+ 39us) server_negotiation.cc:407] Connection header received
0924 03:01:45.414959 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.414960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.414972 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.415047 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.415422 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37188: BlockingRecv error: recv got EOF from 127.0.0.1:37188 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
W20260924 03:01:45.481307 23070 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:45.480588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37202 (local address 127.0.98.61:42549)
0924 03:01:45.480691 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.480694 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.480765 (+ 71us) server_negotiation.cc:407] Connection header received
0924 03:01:45.480846 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.480847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.480858 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.480905 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.481245 (+ 340us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37202: BlockingRecv error: recv got EOF from 127.0.0.1:37202 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260924 03:01:45.611018 23070 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:45.610333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37204 (local address 127.0.98.61:42549)
0924 03:01:45.610385 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.610387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.610458 (+ 71us) server_negotiation.cc:407] Connection header received
0924 03:01:45.610559 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.610561 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.610571 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.610615 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.610952 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37204: BlockingRecv error: recv got EOF from 127.0.0.1:37204 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260924 03:01:45.868745 23070 negotiation.cc:337] Failed RPC negotiation. Trace:
0924 03:01:45.868106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37212 (local address 127.0.98.61:42549)
0924 03:01:45.868165 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:45.868168 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:45.868179 (+ 11us) server_negotiation.cc:407] Connection header received
0924 03:01:45.868262 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:45.868263 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:45.868275 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:45.868339 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0924 03:01:45.868686 (+ 347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37212: BlockingRecv error: recv got EOF from 127.0.0.1:37212 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
I20260924 03:01:46.206664 23158 catalog_manager.cc:1259] Loaded cluster ID: e23d85fd39494b94b29205d6b231a9f7
I20260924 03:01:46.206745 23158 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da: loading cluster ID for follower catalog manager: success
I20260924 03:01:46.207588 23158 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da: acquiring CA information for follower catalog manager: success
I20260924 03:01:46.207872 23158 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9e74069e7ae3413eb24e0eb8b92ec6da: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:01:46.284574 23204 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b: acquiring CA information for follower catalog manager: success
I20260924 03:01:46.284879 23204 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 569e3df64bdb4ca6b6009feae1a5607b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260924 03:01:46.384639 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:46.381956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37220 (local address 127.0.98.61:42549)
0924 03:01:46.382060 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:46.382062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:46.382110 (+ 48us) server_negotiation.cc:407] Connection header received
0924 03:01:46.382173 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:46.382174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:46.382224 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:46.382298 (+ 74us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:01:46.382430 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:46.382989 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:46.383837 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:46.384041 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:46.384104 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:46.384271 (+ 167us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:01:46.384414 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:01:46.384474 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:46.384544 (+ 70us) server_negotiation.cc:299] Negotiation successful
0924 03:01:46.384563 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":58,"threads_started":1}
I20260924 03:01:46.388912 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:46.386419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34978 (local address 127.0.98.60:33243)
0924 03:01:46.386584 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:46.386586 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:46.386595 (+ 9us) server_negotiation.cc:407] Connection header received
0924 03:01:46.386638 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:46.386639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:46.386679 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:46.386754 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:01:46.386837 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:46.387321 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:46.388082 (+ 761us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:46.388284 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:46.388344 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:46.388536 (+ 192us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:01:46.388662 (+ 126us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:01:46.388766 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:46.388828 (+ 62us) server_negotiation.cc:299] Negotiation successful
0924 03:01:46.388849 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":63,"threads_started":1}
I20260924 03:01:46.393038 23276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:46.390456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37228 (local address 127.0.98.61:42549)
0924 03:01:46.390546 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:46.390548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:46.390654 (+ 106us) server_negotiation.cc:407] Connection header received
0924 03:01:46.390775 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:46.390776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:46.390789 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:46.390817 (+ 28us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:01:46.390977 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:46.391467 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:46.392353 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:46.392649 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:46.392708 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:46.392821 (+ 113us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:01:46.392852 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:01:46.392894 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:46.392942 (+ 48us) server_negotiation.cc:299] Negotiation successful
0924 03:01:46.392963 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260924 03:01:46.393500 23282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:46.390707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34986 (local address 127.0.98.60:33243)
0924 03:01:46.390754 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:46.390756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:46.390811 (+ 55us) server_negotiation.cc:407] Connection header received
0924 03:01:46.390865 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:46.390866 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:46.390875 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:46.390922 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:01:46.391043 (+ 121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:46.391765 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:46.392656 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:46.392924 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:46.392974 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:46.393274 (+ 300us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:01:46.393307 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:01:46.393350 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:46.393413 (+ 63us) server_negotiation.cc:299] Negotiation successful
0924 03:01:46.393435 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260924 03:01:46.393635 23289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0924 03:01:46.390431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40914 (local address 127.0.98.62:46431)
0924 03:01:46.390547 (+ 116us) server_negotiation.cc:206] Beginning negotiation
0924 03:01:46.390550 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0924 03:01:46.390654 (+ 104us) server_negotiation.cc:407] Connection header received
0924 03:01:46.390704 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0924 03:01:46.390705 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0924 03:01:46.390756 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0924 03:01:46.390829 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0924 03:01:46.390987 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:46.391765 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:46.392634 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0924 03:01:46.392922 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0924 03:01:46.392977 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0924 03:01:46.393207 (+ 230us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0924 03:01:46.393314 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0924 03:01:46.393365 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0924 03:01:46.393519 (+ 154us) server_negotiation.cc:299] Negotiation successful
0924 03:01:46.393546 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":53,"threads_started":1}
I20260924 03:01:46.395223 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 22923
I20260924 03:01:46.399935 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 22998
I20260924 03:01:46.404031 392 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskPoI4uN/build/release/bin/kudu with pid 23073
2026-09-24T03:01:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2670 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5341 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (259074 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST